[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:01:56 dist-test-slave-j368 krb5kdc[6240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:01:56 dist-test-slave-j368 krb5kdc[6240](info): set up 2 sockets
Jan 31 03:01:56 dist-test-slave-j368 krb5kdc[6240](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260131 03:01:58.386183 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.799s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:01:58 dist-test-slave-j368 krb5kdc[6240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:01:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:01:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:01:58.511850 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:38867
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:01:58.611505 6256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:01:58.611800 6256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:01:58.611922 6256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:01:58.616709 6256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:01:58.616874 6256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:01:58.616922 6256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:01:58.616967 6256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:01:58.617005 6256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:01:58.621982 6256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:38867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:38867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:01:58.623119 6256 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:01:58.623955 6256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:01:58.629714 6261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:01:58.629716 6262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:01:58.629756 6256 server_base.cc:1060] running on GCE node
W20260131 03:01:58.629894 6264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:01:58.630375 6256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:01:58.631258 6256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:01:58.632405 6256 hybrid_clock.cc:648] HybridClock initialized: now 1769828518632389 us; error 31 us; skew 500 ppm
Jan 31 03:01:58 dist-test-slave-j368 krb5kdc[6240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:01:58.634884 6256 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:01:58.635866 6256 webserver.cc:492] Webserver started at http://127.6.22.126:44253/ using document root <none> and password file <none>
I20260131 03:01:58.636385 6256 fs_manager.cc:362] Metadata directory not provided
I20260131 03:01:58.636468 6256 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:01:58.636729 6256 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:01:58.638401 6256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "4064cf10df87478498feefeca80e5a25"
format_stamp: "Formatted at 2026-01-31 03:01:58 on dist-test-slave-j368"
server_key: "e6adb0cd5b8a4b3128bb82214fe9e47a"
server_key_iv: "213fbe20c4f295605db1b9202dc20a62"
server_key_version: "encryptionkey@0"
I20260131 03:01:58.638906 6256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "4064cf10df87478498feefeca80e5a25"
format_stamp: "Formatted at 2026-01-31 03:01:58 on dist-test-slave-j368"
server_key: "e6adb0cd5b8a4b3128bb82214fe9e47a"
server_key_iv: "213fbe20c4f295605db1b9202dc20a62"
server_key_version: "encryptionkey@0"
I20260131 03:01:58.642131 6256 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260131 03:01:58.644209 6271 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:01:58.645156 6256 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:01:58.645308 6256 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "4064cf10df87478498feefeca80e5a25"
format_stamp: "Formatted at 2026-01-31 03:01:58 on dist-test-slave-j368"
server_key: "e6adb0cd5b8a4b3128bb82214fe9e47a"
server_key_iv: "213fbe20c4f295605db1b9202dc20a62"
server_key_version: "encryptionkey@0"
I20260131 03:01:58.645412 6256 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:01:58.673411 6256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:01:58.676230 6256 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:01:58.676417 6256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:01:58.683372 6323 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:38867 every 8 connection(s)
I20260131 03:01:58.683377 6256 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:38867
I20260131 03:01:58.684316 6256 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:01:58.687000 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 6256
I20260131 03:01:58.687150 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:01:58.687348 6324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:01:58.688354 6233 external_mini_cluster.cc:1468] Setting key cc879ae771a0611b0291a80b65c3ce50
I20260131 03:01:58.694094 6324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25: Bootstrap starting.
I20260131 03:01:58.696382 6324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25: Neither blocks nor log segments found. Creating new log.
I20260131 03:01:58.697230 6324 log.cc:826] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25: Log is configured to *not* fsync() on all Append() calls
I20260131 03:01:58.699213 6324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25: No bootstrap required, opened a new log
Jan 31 03:01:58 dist-test-slave-j368 krb5kdc[6240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:01:58.702317 6324 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4064cf10df87478498feefeca80e5a25" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38867 } }
I20260131 03:01:58.702466 6324 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:01:58.702521 6324 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4064cf10df87478498feefeca80e5a25, State: Initialized, Role: FOLLOWER
I20260131 03:01:58.702986 6324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4064cf10df87478498feefeca80e5a25" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38867 } }
I20260131 03:01:58.703146 6324 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:01:58.703220 6324 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:01:58.703331 6324 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:01:58.704079 6324 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4064cf10df87478498feefeca80e5a25" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38867 } }
I20260131 03:01:58.704408 6324 leader_election.cc:304] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4064cf10df87478498feefeca80e5a25; no voters:
I20260131 03:01:58.704753 6324 leader_election.cc:290] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:01:58.704802 6329 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:01:58.704969 6329 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [term 1 LEADER]: Becoming Leader. State: Replica: 4064cf10df87478498feefeca80e5a25, State: Running, Role: LEADER
I20260131 03:01:58.705299 6329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4064cf10df87478498feefeca80e5a25" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38867 } }
I20260131 03:01:58.705794 6324 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:01:58.706519 6329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4064cf10df87478498feefeca80e5a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4064cf10df87478498feefeca80e5a25" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38867 } } }
I20260131 03:01:58.706715 6329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [sys.catalog]: This master's current role is: LEADER
I20260131 03:01:58.708076 6337 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:01:58.709465 6330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4064cf10df87478498feefeca80e5a25. Latest consensus state: current_term: 1 leader_uuid: "4064cf10df87478498feefeca80e5a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4064cf10df87478498feefeca80e5a25" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38867 } } }
I20260131 03:01:58.709596 6330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25 [sys.catalog]: This master's current role is: LEADER
I20260131 03:01:58.709985 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:01:58.691196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.6.22.126:38867)
0131 03:01:58.691672 (+ 476us) server_negotiation.cc:207] Beginning negotiation
0131 03:01:58.691690 (+ 18us) server_negotiation.cc:400] Waiting for connection header
0131 03:01:58.692006 (+ 316us) server_negotiation.cc:408] Connection header received
0131 03:01:58.694710 (+ 2704us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:01:58.694724 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:01:58.694993 (+ 269us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:01:58.695227 (+ 234us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:01:58.697447 (+ 2220us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:01:58.698178 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:01:58.699136 (+ 958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:01:58.699392 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:01:58.702832 (+ 3440us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:01:58.702848 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:01:58.702857 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:01:58.702887 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:01:58.704928 (+ 2041us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:01:58.705506 (+ 578us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:01:58.705511 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:01:58.705516 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:01:58.705593 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:01:58.705886 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:01:58.705890 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:01:58.705892 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:01:58.706297 (+ 405us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:01:58.708829 (+ 2532us) server_negotiation.cc:1036] Waiting for connection context
0131 03:01:58.708979 (+ 150us) server_negotiation.cc:300] Negotiation successful
0131 03:01:58.709217 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":114,"threads_started":1}
I20260131 03:01:58.710978 6337 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:01:58.716358 6337 catalog_manager.cc:1357] Generated new cluster ID: 93348493f0df4f6ebd0a8b674b0fdca4
I20260131 03:01:58.716423 6337 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:01:58.747798 6337 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:01:58.748644 6337 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:01:58.753247 6337 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 4064cf10df87478498feefeca80e5a25: Generated new TSK 0
I20260131 03:01:58.754022 6337 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:01:58.801719 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:38867
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:01:58.903535 6352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:01:58.903715 6352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:01:58.903770 6352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:01:58.906994 6352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:01:58.907044 6352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:01:58.907178 6352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:01:58.911048 6352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:38867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:01:58.911955 6352 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:01:58.912793 6352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:01:58.918663 6358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:01:58.918666 6357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:01:58.918685 6360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:01:58.918861 6352 server_base.cc:1060] running on GCE node
I20260131 03:01:58.919325 6352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:01:58.919826 6352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:01:58.920964 6352 hybrid_clock.cc:648] HybridClock initialized: now 1769828518920944 us; error 34 us; skew 500 ppm
Jan 31 03:01:58 dist-test-slave-j368 krb5kdc[6240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:01:58.923255 6352 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:01:58.924190 6352 webserver.cc:492] Webserver started at http://127.6.22.65:38325/ using document root <none> and password file <none>
I20260131 03:01:58.924710 6352 fs_manager.cc:362] Metadata directory not provided
I20260131 03:01:58.924810 6352 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:01:58.924962 6352 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:01:58.926486 6352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "7760c55d1272466a8198b6c436d833aa"
format_stamp: "Formatted at 2026-01-31 03:01:58 on dist-test-slave-j368"
server_key: "945e7bc2e7f42c8b3be782c1e7f79349"
server_key_iv: "3d7b20381b328955e964abf5b34208fa"
server_key_version: "encryptionkey@0"
I20260131 03:01:58.926906 6352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "7760c55d1272466a8198b6c436d833aa"
format_stamp: "Formatted at 2026-01-31 03:01:58 on dist-test-slave-j368"
server_key: "945e7bc2e7f42c8b3be782c1e7f79349"
server_key_iv: "3d7b20381b328955e964abf5b34208fa"
server_key_version: "encryptionkey@0"
I20260131 03:01:58.930059 6352 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260131 03:01:58.932056 6367 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:01:58.933063 6352 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:01:58.933182 6352 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "7760c55d1272466a8198b6c436d833aa"
format_stamp: "Formatted at 2026-01-31 03:01:58 on dist-test-slave-j368"
server_key: "945e7bc2e7f42c8b3be782c1e7f79349"
server_key_iv: "3d7b20381b328955e964abf5b34208fa"
server_key_version: "encryptionkey@0"
I20260131 03:01:58.933276 6352 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:01:58.946048 6352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:01:58.948781 6352 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:01:58.948959 6352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:01:58.949501 6352 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:01:58.950358 6352 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:01:58.950428 6352 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:01:58.950489 6352 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:01:58.950532 6352 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:01:58.959404 6352 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:35937
I20260131 03:01:58.959435 6480 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:35937 every 8 connection(s)
I20260131 03:01:58.960260 6352 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:01:58.966390 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 6352
I20260131 03:01:58.966485 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:01:58.966701 6233 external_mini_cluster.cc:1468] Setting key be7451e8cdde06a111cda8ebcdddb963
Jan 31 03:01:58 dist-test-slave-j368 krb5kdc[6240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:01:58.972086 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:01:58.961860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:48861 (local address 127.6.22.126:38867)
0131 03:01:58.962008 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0131 03:01:58.962013 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:01:58.962603 (+ 590us) server_negotiation.cc:408] Connection header received
0131 03:01:58.963330 (+ 727us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:01:58.963333 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:01:58.963380 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:01:58.963453 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:01:58.964989 (+ 1536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:01:58.965492 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:01:58.966093 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:01:58.966229 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:01:58.968382 (+ 2153us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:01:58.968397 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:01:58.968399 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:01:58.968422 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:01:58.970145 (+ 1723us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:01:58.970609 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:01:58.970616 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:01:58.970620 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:01:58.970680 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:01:58.970974 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:01:58.970980 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:01:58.970985 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:01:58.971177 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:01:58.971284 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0131 03:01:58.971790 (+ 506us) server_negotiation.cc:300] Negotiation successful
0131 03:01:58.971927 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260131 03:01:58.972492 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:01:58.962082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38867 (local address 127.6.22.65:48861)
0131 03:01:58.962445 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0131 03:01:58.962479 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:01:58.963162 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:01:58.963600 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:01:58.963607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:01:58.963944 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:01:58.964856 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:01:58.964869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:01:58.965593 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:01:58.965596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:01:58.965998 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:01:58.966004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:01:58.966152 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:01:58.966617 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:01:58.966638 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:01:58.968220 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:01:58.970253 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:01:58.970257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:01:58.970263 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:01:58.970487 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:01:58.970789 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:01:58.970791 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:01:58.970792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:01:58.970879 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:01:58.971279 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:01:58.971284 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:01:58.971509 (+ 225us) client_negotiation.cc:769] Sending connection context
0131 03:01:58.971701 (+ 192us) client_negotiation.cc:240] Negotiation successful
0131 03:01:58.971883 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
I20260131 03:01:58.973585 6481 heartbeater.cc:344] Connected to a master server at 127.6.22.126:38867
I20260131 03:01:58.973820 6481 heartbeater.cc:461] Registering TS with master...
I20260131 03:01:58.974411 6481 heartbeater.cc:507] Master 127.6.22.126:38867 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
I20260131 03:01:58.975867 6288 ts_manager.cc:194] Registered new tserver with Master: 7760c55d1272466a8198b6c436d833aa (127.6.22.65:35937)
I20260131 03:01:58.977090 6288 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:48861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:01:59.012151 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:38867
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:01:59.112612 6488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:01:59.112810 6488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:01:59.112862 6488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:01:59.115842 6488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:01:59.115888 6488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:01:59.115960 6488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:01:59.119849 6488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:38867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:01:59.120802 6488 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:01:59.121538 6488 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:01:59.127625 6496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:01:59.127648 6493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:01:59.127692 6494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:01:59.127943 6488 server_base.cc:1060] running on GCE node
I20260131 03:01:59.128283 6488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:01:59.128903 6488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:01:59.130075 6488 hybrid_clock.cc:648] HybridClock initialized: now 1769828519130061 us; error 27 us; skew 500 ppm
Jan 31 03:01:59 dist-test-slave-j368 krb5kdc[6240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828519, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:01:59.132447 6488 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:01:59.133484 6488 webserver.cc:492] Webserver started at http://127.6.22.66:39479/ using document root <none> and password file <none>
I20260131 03:01:59.134001 6488 fs_manager.cc:362] Metadata directory not provided
I20260131 03:01:59.134078 6488 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:01:59.134276 6488 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:01:59.135879 6488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "5c801208eb114b3d9c60d6f03707a019"
format_stamp: "Formatted at 2026-01-31 03:01:59 on dist-test-slave-j368"
server_key: "16f8cf12367eb59dab627d3f80339533"
server_key_iv: "aa62ee8c427225e89264aeca92117ad9"
server_key_version: "encryptionkey@0"
I20260131 03:01:59.136332 6488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "5c801208eb114b3d9c60d6f03707a019"
format_stamp: "Formatted at 2026-01-31 03:01:59 on dist-test-slave-j368"
server_key: "16f8cf12367eb59dab627d3f80339533"
server_key_iv: "aa62ee8c427225e89264aeca92117ad9"
server_key_version: "encryptionkey@0"
I20260131 03:01:59.139338 6488 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260131 03:01:59.141333 6503 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:01:59.142298 6488 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260131 03:01:59.142416 6488 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "5c801208eb114b3d9c60d6f03707a019"
format_stamp: "Formatted at 2026-01-31 03:01:59 on dist-test-slave-j368"
server_key: "16f8cf12367eb59dab627d3f80339533"
server_key_iv: "aa62ee8c427225e89264aeca92117ad9"
server_key_version: "encryptionkey@0"
I20260131 03:01:59.142506 6488 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:01:59.152815 6488 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:01:59.155154 6488 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:01:59.155323 6488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:01:59.155841 6488 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:01:59.156628 6488 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:01:59.156754 6488 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:01:59.156827 6488 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:01:59.156885 6488 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:01:59.165632 6488 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:44485
I20260131 03:01:59.165665 6616 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:44485 every 8 connection(s)
I20260131 03:01:59.166491 6488 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:01:59 dist-test-slave-j368 krb5kdc[6240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828519, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:01:59.176684 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 6488
I20260131 03:01:59.176805 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:01:59.177059 6233 external_mini_cluster.cc:1468] Setting key 3cd2e5381c549fb781485715aa19bf19
I20260131 03:01:59.177942 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:01:59.168060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:53173 (local address 127.6.22.126:38867)
0131 03:01:59.168228 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0131 03:01:59.168233 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:01:59.168876 (+ 643us) server_negotiation.cc:408] Connection header received
0131 03:01:59.169618 (+ 742us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:01:59.169621 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:01:59.169668 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:01:59.169747 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:01:59.171307 (+ 1560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:01:59.171994 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:01:59.172580 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:01:59.172878 (+ 298us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:01:59.174692 (+ 1814us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:01:59.174709 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:01:59.174711 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:01:59.174734 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:01:59.175976 (+ 1242us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:01:59.176397 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:01:59.176400 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:01:59.176401 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:01:59.176441 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:01:59.176861 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:01:59.176865 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:01:59.176869 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:01:59.177021 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:01:59.177101 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0131 03:01:59.177671 (+ 570us) server_negotiation.cc:300] Negotiation successful
0131 03:01:59.177765 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260131 03:01:59.178629 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:01:59.168342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38867 (local address 127.6.22.66:53173)
0131 03:01:59.168750 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0131 03:01:59.168782 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0131 03:01:59.169451 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:01:59.169885 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:01:59.169893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:01:59.170249 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:01:59.171158 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:01:59.171172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:01:59.172102 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:01:59.172104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:01:59.172481 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:01:59.172487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:01:59.172637 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:01:59.173213 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:01:59.173235 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:01:59.174548 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:01:59.176082 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:01:59.176086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:01:59.176094 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:01:59.176311 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:01:59.176541 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:01:59.176546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:01:59.176548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:01:59.176718 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:01:59.177132 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:01:59.177138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:01:59.177392 (+ 254us) client_negotiation.cc:769] Sending connection context
0131 03:01:59.177583 (+ 191us) client_negotiation.cc:240] Negotiation successful
0131 03:01:59.177855 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":110,"threads_started":1}
I20260131 03:01:59.179643 6617 heartbeater.cc:344] Connected to a master server at 127.6.22.126:38867
I20260131 03:01:59.179880 6617 heartbeater.cc:461] Registering TS with master...
I20260131 03:01:59.180310 6617 heartbeater.cc:507] Master 127.6.22.126:38867 requested a full tablet report, sending...
I20260131 03:01:59.181219 6288 ts_manager.cc:194] Registered new tserver with Master: 5c801208eb114b3d9c60d6f03707a019 (127.6.22.66:44485)
I20260131 03:01:59.181795 6288 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:53173
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:01:59.221339 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:38867
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:01:59.323386 6624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:01:59.323554 6624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:01:59.323676 6624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:01:59.327003 6624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:01:59.327054 6624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:01:59.327180 6624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:01:59.331287 6624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:38867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:01:59.332172 6624 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:01:59.332966 6624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:01:59.339145 6629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:01:59.339141 6632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:01:59.339160 6630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:01:59.339251 6624 server_base.cc:1060] running on GCE node
I20260131 03:01:59.339708 6624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:01:59.340202 6624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:01:59.341398 6624 hybrid_clock.cc:648] HybridClock initialized: now 1769828519341376 us; error 41 us; skew 500 ppm
Jan 31 03:01:59 dist-test-slave-j368 krb5kdc[6240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828519, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:01:59.343890 6624 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:01:59.344939 6624 webserver.cc:492] Webserver started at http://127.6.22.67:43157/ using document root <none> and password file <none>
I20260131 03:01:59.345402 6624 fs_manager.cc:362] Metadata directory not provided
I20260131 03:01:59.345460 6624 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:01:59.345605 6624 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:01:59.347152 6624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "245604a949de44449aa1bf87c59dc6b3"
format_stamp: "Formatted at 2026-01-31 03:01:59 on dist-test-slave-j368"
server_key: "5d1fe4954503e80fbd20b0e1b4f47c7d"
server_key_iv: "52cf56b82a6c389800a168a0782b3219"
server_key_version: "encryptionkey@0"
I20260131 03:01:59.347594 6624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "245604a949de44449aa1bf87c59dc6b3"
format_stamp: "Formatted at 2026-01-31 03:01:59 on dist-test-slave-j368"
server_key: "5d1fe4954503e80fbd20b0e1b4f47c7d"
server_key_iv: "52cf56b82a6c389800a168a0782b3219"
server_key_version: "encryptionkey@0"
I20260131 03:01:59.350816 6624 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260131 03:01:59.352648 6639 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:01:59.353654 6624 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:01:59.353763 6624 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "245604a949de44449aa1bf87c59dc6b3"
format_stamp: "Formatted at 2026-01-31 03:01:59 on dist-test-slave-j368"
server_key: "5d1fe4954503e80fbd20b0e1b4f47c7d"
server_key_iv: "52cf56b82a6c389800a168a0782b3219"
server_key_version: "encryptionkey@0"
I20260131 03:01:59.353858 6624 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:01:59.382279 6624 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:01:59.385048 6624 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:01:59.385224 6624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:01:59.385696 6624 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:01:59.386500 6624 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:01:59.386543 6624 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:01:59.386605 6624 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:01:59.386638 6624 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:01:59.396136 6624 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:39541
I20260131 03:01:59.396164 6752 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:39541 every 8 connection(s)
I20260131 03:01:59.397231 6624 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:01:59 dist-test-slave-j368 krb5kdc[6240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828519, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:01:59.406045 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 6624
I20260131 03:01:59.406160 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:01:59.406379 6233 external_mini_cluster.cc:1468] Setting key 7735cebf6f29c225970a9acb9ede5657
I20260131 03:01:59.409389 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:01:59.398933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:48669 (local address 127.6.22.126:38867)
0131 03:01:59.399077 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0131 03:01:59.399081 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:01:59.399699 (+ 618us) server_negotiation.cc:408] Connection header received
0131 03:01:59.400693 (+ 994us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:01:59.400697 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:01:59.400765 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:01:59.400859 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:01:59.402420 (+ 1561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:01:59.402926 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:01:59.403596 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:01:59.403786 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:01:59.405976 (+ 2190us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:01:59.405993 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:01:59.405995 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:01:59.406018 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:01:59.407559 (+ 1541us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:01:59.408044 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:01:59.408047 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:01:59.408048 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:01:59.408095 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:01:59.408405 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:01:59.408408 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:01:59.408411 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:01:59.408613 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:01:59.408760 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
0131 03:01:59.409189 (+ 429us) server_negotiation.cc:300] Negotiation successful
0131 03:01:59.409280 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260131 03:01:59.410027 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:01:59.399177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38867 (local address 127.6.22.67:48669)
0131 03:01:59.399568 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0131 03:01:59.399601 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:01:59.400345 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:01:59.401021 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:01:59.401030 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:01:59.401376 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:01:59.402269 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:01:59.402283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:01:59.403045 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:01:59.403050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:01:59.403476 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:01:59.403483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:01:59.403699 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:01:59.404269 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:01:59.404298 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:01:59.405795 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:01:59.407668 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:01:59.407680 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:01:59.407690 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:01:59.407956 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:01:59.408185 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:01:59.408188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:01:59.408190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:01:59.408310 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:01:59.408752 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:01:59.408758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:01:59.408983 (+ 225us) client_negotiation.cc:769] Sending connection context
0131 03:01:59.409140 (+ 157us) client_negotiation.cc:240] Negotiation successful
0131 03:01:59.409380 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":114,"threads_started":1}
I20260131 03:01:59.411001 6753 heartbeater.cc:344] Connected to a master server at 127.6.22.126:38867
I20260131 03:01:59.411239 6753 heartbeater.cc:461] Registering TS with master...
I20260131 03:01:59.411718 6753 heartbeater.cc:507] Master 127.6.22.126:38867 requested a full tablet report, sending...
I20260131 03:01:59.412627 6288 ts_manager.cc:194] Registered new tserver with Master: 245604a949de44449aa1bf87c59dc6b3 (127.6.22.67:39541)
I20260131 03:01:59.413187 6288 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:48669
I20260131 03:01:59.420286 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 31 03:01:59 dist-test-slave-j368 krb5kdc[6240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 31 03:01:59 dist-test-slave-j368 krb5kdc[6240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.22.65@KRBTEST.COM
I20260131 03:01:59.442431 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:01:59.434836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.6.22.65:35937)
0131 03:01:59.435138 (+ 302us) server_negotiation.cc:207] Beginning negotiation
0131 03:01:59.435145 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:01:59.435178 (+ 33us) server_negotiation.cc:408] Connection header received
0131 03:01:59.435226 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:01:59.435231 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:01:59.435381 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:01:59.435494 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:01:59.436227 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:01:59.436917 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:01:59.437545 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:01:59.437734 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:01:59.439642 (+ 1908us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:01:59.439661 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:01:59.439670 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:01:59.439693 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:01:59.441204 (+ 1511us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:01:59.441530 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:01:59.441534 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:01:59.441535 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:01:59.441579 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:01:59.441746 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:01:59.441748 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:01:59.441749 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:01:59.441895 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:01:59.442027 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0131 03:01:59.442173 (+ 146us) server_negotiation.cc:300] Negotiation successful
0131 03:01:59.442306 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":112,"threads_started":1}
W20260131 03:01:59.443262 6395 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42028
Jan 31 03:01:59 dist-test-slave-j368 krb5kdc[6240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 31 03:01:59 dist-test-slave-j368 krb5kdc[6240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.65@KRBTEST.COM
I20260131 03:01:59.465150 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:01:59.458251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.6.22.65:35937)
0131 03:01:59.458383 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0131 03:01:59.458387 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:01:59.458493 (+ 106us) server_negotiation.cc:408] Connection header received
0131 03:01:59.458575 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:01:59.458577 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:01:59.458616 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:01:59.458683 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:01:59.459352 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:01:59.459809 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:01:59.460377 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:01:59.460534 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:01:59.462508 (+ 1974us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:01:59.462523 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:01:59.462525 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:01:59.462546 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:01:59.463944 (+ 1398us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:01:59.464251 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:01:59.464254 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:01:59.464255 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:01:59.464291 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:01:59.464452 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:01:59.464453 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:01:59.464454 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:01:59.464550 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:01:59.464635 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0131 03:01:59.464863 (+ 228us) server_negotiation.cc:300] Negotiation successful
0131 03:01:59.464944 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260131 03:01:59.466006 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 6352
I20260131 03:01:59.472028 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 6488
I20260131 03:01:59.479404 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 6624
I20260131 03:01:59.484617 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 6256
2026-01-31T03:01:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2924 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:01:59 dist-test-slave-j368 krb5kdc[6767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:01:59 dist-test-slave-j368 krb5kdc[6767](info): set up 2 sockets
Jan 31 03:01:59 dist-test-slave-j368 krb5kdc[6767](info): commencing operation
krb5kdc: starting...
W20260131 03:02:01.465981 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.959s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:02:01 dist-test-slave-j368 krb5kdc[6767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:02:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:01.582841 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:41235
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:41235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:01.675169 6783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:01.675328 6783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:01.675372 6783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:01.678099 6783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:01.678143 6783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:01.678162 6783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:01.678177 6783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:01.678192 6783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:01.681770 6783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:41235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:41235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:01.682618 6783 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:01.683351 6783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:01.689258 6791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:01.689263 6788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:01.689474 6783 server_base.cc:1060] running on GCE node
W20260131 03:02:01.689338 6789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:01.690034 6783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:01.690832 6783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:01.691977 6783 hybrid_clock.cc:648] HybridClock initialized: now 1769828521691955 us; error 37 us; skew 500 ppm
Jan 31 03:02:01 dist-test-slave-j368 krb5kdc[6767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828521, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:01.694193 6783 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:02:01.695117 6783 webserver.cc:492] Webserver started at http://127.6.22.126:34725/ using document root <none> and password file <none>
I20260131 03:02:01.695608 6783 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:01.695654 6783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:01.695858 6783 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:01.697377 6783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "6ad781e7472f4f64824a2bc035a0457c"
format_stamp: "Formatted at 2026-01-31 03:02:01 on dist-test-slave-j368"
server_key: "f54ba0360033638f7d1d99680c48ade1"
server_key_iv: "cdf7f3271a67f38c654ddfb36d8e7ae6"
server_key_version: "encryptionkey@0"
I20260131 03:02:01.697816 6783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "6ad781e7472f4f64824a2bc035a0457c"
format_stamp: "Formatted at 2026-01-31 03:02:01 on dist-test-slave-j368"
server_key: "f54ba0360033638f7d1d99680c48ade1"
server_key_iv: "cdf7f3271a67f38c654ddfb36d8e7ae6"
server_key_version: "encryptionkey@0"
I20260131 03:02:01.700809 6783 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:02:01.702646 6798 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:01.703560 6783 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:01.703678 6783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "6ad781e7472f4f64824a2bc035a0457c"
format_stamp: "Formatted at 2026-01-31 03:02:01 on dist-test-slave-j368"
server_key: "f54ba0360033638f7d1d99680c48ade1"
server_key_iv: "cdf7f3271a67f38c654ddfb36d8e7ae6"
server_key_version: "encryptionkey@0"
I20260131 03:02:01.703773 6783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:01.722199 6783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:01.726980 6783 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:01.727128 6783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:01.733268 6850 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:41235 every 8 connection(s)
I20260131 03:02:01.733273 6783 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:41235
I20260131 03:02:01.734082 6783 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:01.736529 6851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:01.736819 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 6783
I20260131 03:02:01.736936 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:01.737152 6233 external_mini_cluster.cc:1468] Setting key df618a1c2a1949a55737b342266287cb
I20260131 03:02:01.742051 6851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c: Bootstrap starting.
Jan 31 03:02:01 dist-test-slave-j368 krb5kdc[6767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:01.744215 6851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:01.744974 6851 log.cc:826] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:01.746791 6851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c: No bootstrap required, opened a new log
I20260131 03:02:01.748201 6854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:01.738219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.6.22.126:41235)
0131 03:02:01.738605 (+ 386us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:01.738613 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:01.738641 (+ 28us) server_negotiation.cc:408] Connection header received
0131 03:02:01.739274 (+ 633us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:01.739292 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:01.739542 (+ 250us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:01.739837 (+ 295us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:01.740698 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:01.741372 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:01.741975 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:01.742257 (+ 282us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:01.744332 (+ 2075us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:01.744346 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:01.744354 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:01.744380 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:01.746084 (+ 1704us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:01.746491 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:01.746494 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:01.746498 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:01.746565 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:01.746778 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:01.746780 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:01.746781 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:01.747070 (+ 289us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:01.747198 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:01.747474 (+ 276us) server_negotiation.cc:300] Negotiation successful
0131 03:02:01.747697 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":141,"threads_started":1}
I20260131 03:02:01.749213 6851 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad781e7472f4f64824a2bc035a0457c" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41235 } }
I20260131 03:02:01.749409 6851 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:01.749475 6851 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad781e7472f4f64824a2bc035a0457c, State: Initialized, Role: FOLLOWER
I20260131 03:02:01.749884 6851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad781e7472f4f64824a2bc035a0457c" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41235 } }
I20260131 03:02:01.750037 6851 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:01.750100 6851 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:01.750193 6851 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:01.751231 6851 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad781e7472f4f64824a2bc035a0457c" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41235 } }
I20260131 03:02:01.751562 6851 leader_election.cc:304] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ad781e7472f4f64824a2bc035a0457c; no voters:
I20260131 03:02:01.751807 6851 leader_election.cc:290] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:01.751919 6856 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:01.752183 6856 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [term 1 LEADER]: Becoming Leader. State: Replica: 6ad781e7472f4f64824a2bc035a0457c, State: Running, Role: LEADER
I20260131 03:02:01.752429 6856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad781e7472f4f64824a2bc035a0457c" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41235 } }
I20260131 03:02:01.752804 6851 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:01.754379 6857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ad781e7472f4f64824a2bc035a0457c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad781e7472f4f64824a2bc035a0457c" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41235 } } }
I20260131 03:02:01.754495 6857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:01.754382 6858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ad781e7472f4f64824a2bc035a0457c. Latest consensus state: current_term: 1 leader_uuid: "6ad781e7472f4f64824a2bc035a0457c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad781e7472f4f64824a2bc035a0457c" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41235 } } }
I20260131 03:02:01.755077 6858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c [sys.catalog]: This master's current role is: LEADER
W20260131 03:02:01.756747 6870 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:02:01.756799 6870 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:02:01.756862 6871 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:01.757478 6871 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:01.761866 6871 catalog_manager.cc:1357] Generated new cluster ID: 45368a8bb5984aae9b9f893c7bfcb15b
I20260131 03:02:01.761941 6871 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:01.775755 6871 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:01.776734 6871 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:01.794910 6871 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 6ad781e7472f4f64824a2bc035a0457c: Generated new TSK 0
I20260131 03:02:01.795504 6871 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:01.837977 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:41235
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:01.931303 6879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:01.931473 6879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:01.931562 6879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:01.934337 6879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:01.934381 6879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:01.934486 6879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:02:01.938292 6879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:41235
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:01.939138 6879 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:01.939913 6879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:01.945607 6885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:01.945640 6879 server_base.cc:1060] running on GCE node
W20260131 03:02:01.945570 6887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:01.945575 6884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:01.946075 6879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:01.946547 6879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:01.947705 6879 hybrid_clock.cc:648] HybridClock initialized: now 1769828521947694 us; error 25 us; skew 500 ppm
Jan 31 03:02:01 dist-test-slave-j368 krb5kdc[6767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828521, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:01.949990 6879 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:02:01.950887 6879 webserver.cc:492] Webserver started at http://127.6.22.65:38869/ using document root <none> and password file <none>
I20260131 03:02:01.951342 6879 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:01.951406 6879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:01.951572 6879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:01.952992 6879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "4d917c6a566a4c6d895ab8477f2c4a9c"
format_stamp: "Formatted at 2026-01-31 03:02:01 on dist-test-slave-j368"
server_key: "59e25410080da63618c38d976e01871b"
server_key_iv: "acb3eaa2b7b859ebdd28bc15f27652a6"
server_key_version: "encryptionkey@0"
I20260131 03:02:01.953404 6879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "4d917c6a566a4c6d895ab8477f2c4a9c"
format_stamp: "Formatted at 2026-01-31 03:02:01 on dist-test-slave-j368"
server_key: "59e25410080da63618c38d976e01871b"
server_key_iv: "acb3eaa2b7b859ebdd28bc15f27652a6"
server_key_version: "encryptionkey@0"
I20260131 03:02:01.956286 6879 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:02:01.958071 6894 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:01.958884 6879 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:02:01.958994 6879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "4d917c6a566a4c6d895ab8477f2c4a9c"
format_stamp: "Formatted at 2026-01-31 03:02:01 on dist-test-slave-j368"
server_key: "59e25410080da63618c38d976e01871b"
server_key_iv: "acb3eaa2b7b859ebdd28bc15f27652a6"
server_key_version: "encryptionkey@0"
I20260131 03:02:01.959081 6879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:01.971614 6879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:01.973878 6879 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:01.974032 6879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:01.974459 6879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:01.975152 6879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:01.975212 6879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:01.975267 6879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:01.975303 6879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:01.983759 6879 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:45751
I20260131 03:02:01.983778 7007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:45751 every 8 connection(s)
I20260131 03:02:01.984527 6879 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:02:01 dist-test-slave-j368 krb5kdc[6767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828521, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:01.992218 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 6879
I20260131 03:02:01.992300 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:02:01.992509 6233 external_mini_cluster.cc:1468] Setting key 73c87e3a22278c1c32e9a7bd442bad31
I20260131 03:02:01.995793 6854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:01.985940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:60415 (local address 127.6.22.126:41235)
0131 03:02:01.986062 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:01.986066 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:01.986591 (+ 525us) server_negotiation.cc:408] Connection header received
0131 03:02:01.987420 (+ 829us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:01.987423 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:01.987471 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:01.987544 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:01.989026 (+ 1482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:01.989544 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:01.990116 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:01.990253 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:01.992116 (+ 1863us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:01.992132 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:01.992135 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:01.992158 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:01.993677 (+ 1519us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:01.994255 (+ 578us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:01.994257 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:01.994259 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:01.994313 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:01.994638 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:01.994640 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:01.994641 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:01.994794 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:01.994865 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:01.995439 (+ 574us) server_negotiation.cc:300] Negotiation successful
0131 03:02:01.995590 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260131 03:02:01.996341 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:01.986131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41235 (local address 127.6.22.65:60415)
0131 03:02:01.986470 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:01.986499 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:01.987253 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:01.987685 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:01.987692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:01.988015 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:01.988912 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:01.988925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:01.989642 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:01.989644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:01.990025 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:01.990031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:01.990197 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:01.990687 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:01.990721 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:01.991975 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:01.993803 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:01.993807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:01.993814 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:01.994121 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:01.994397 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:01.994400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:01.994405 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:01.994546 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:01.994896 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:01.994918 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:01.995202 (+ 284us) client_negotiation.cc:769] Sending connection context
0131 03:02:01.995411 (+ 209us) client_negotiation.cc:240] Negotiation successful
0131 03:02:01.995661 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20260131 03:02:01.997308 7008 heartbeater.cc:344] Connected to a master server at 127.6.22.126:41235
I20260131 03:02:01.997515 7008 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:01.997908 7008 heartbeater.cc:507] Master 127.6.22.126:41235 requested a full tablet report, sending...
I20260131 03:02:01.999300 6815 ts_manager.cc:194] Registered new tserver with Master: 4d917c6a566a4c6d895ab8477f2c4a9c (127.6.22.65:45751)
I20260131 03:02:02.000866 6815 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:60415
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:02.034528 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:41235
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:02.125722 7015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:02.125877 7015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:02.125932 7015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:02.128602 7015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:02.128693 7015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:02.128780 7015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:02:02.132226 7015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:41235
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:02.132969 7015 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:02.133591 7015 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:02.139537 7021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:02.139549 7023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:02.139549 7020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:02.139853 7015 server_base.cc:1060] running on GCE node
I20260131 03:02:02.140174 7015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:02.140708 7015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:02.141844 7015 hybrid_clock.cc:648] HybridClock initialized: now 1769828522141824 us; error 34 us; skew 500 ppm
Jan 31 03:02:02 dist-test-slave-j368 krb5kdc[6767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828522, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:02.143947 7015 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:02:02.144888 7015 webserver.cc:492] Webserver started at http://127.6.22.66:33373/ using document root <none> and password file <none>
I20260131 03:02:02.145303 7015 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:02.145341 7015 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:02.145462 7015 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:02.146775 7015 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "5e7548ab170e4bada22612cf2894212a"
format_stamp: "Formatted at 2026-01-31 03:02:02 on dist-test-slave-j368"
server_key: "9d36b4277100fadc9bcabca4ab933aa2"
server_key_iv: "b03c89b87025e16dd420f24a44d0ff3c"
server_key_version: "encryptionkey@0"
I20260131 03:02:02.147135 7015 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "5e7548ab170e4bada22612cf2894212a"
format_stamp: "Formatted at 2026-01-31 03:02:02 on dist-test-slave-j368"
server_key: "9d36b4277100fadc9bcabca4ab933aa2"
server_key_iv: "b03c89b87025e16dd420f24a44d0ff3c"
server_key_version: "encryptionkey@0"
I20260131 03:02:02.150089 7015 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260131 03:02:02.151811 7030 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:02.152637 7015 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:02.152783 7015 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "5e7548ab170e4bada22612cf2894212a"
format_stamp: "Formatted at 2026-01-31 03:02:02 on dist-test-slave-j368"
server_key: "9d36b4277100fadc9bcabca4ab933aa2"
server_key_iv: "b03c89b87025e16dd420f24a44d0ff3c"
server_key_version: "encryptionkey@0"
I20260131 03:02:02.152861 7015 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:02.169751 7015 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:02.171846 7015 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:02.172011 7015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:02.172453 7015 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:02.173154 7015 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:02.173204 7015 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:02.173257 7015 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:02.173295 7015 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:02.181381 7015 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:36101
I20260131 03:02:02.181407 7143 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:36101 every 8 connection(s)
I20260131 03:02:02.182199 7015 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:02:02.188510 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 7015
I20260131 03:02:02.188596 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:02:02.188939 6233 external_mini_cluster.cc:1468] Setting key b71c9e0d5b2ad0f6b1e0968e81b91088
Jan 31 03:02:02 dist-test-slave-j368 krb5kdc[6767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828522, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:02.193480 6854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.183718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:50165 (local address 127.6.22.126:41235)
0131 03:02:02.183839 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:02.183842 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:02.184419 (+ 577us) server_negotiation.cc:408] Connection header received
0131 03:02:02.185052 (+ 633us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:02.185055 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:02.185103 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:02.185175 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:02.186480 (+ 1305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.186949 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.187464 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.187585 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.189738 (+ 2153us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:02.189756 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:02.189762 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:02.189793 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:02.191442 (+ 1649us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:02.191837 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:02.191843 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:02.191847 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:02.191901 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:02.192362 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:02.192365 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:02.192366 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:02.192496 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:02.192585 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:02.193117 (+ 532us) server_negotiation.cc:300] Negotiation successful
0131 03:02:02.193248 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260131 03:02:02.193739 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.183918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41235 (local address 127.6.22.66:50165)
0131 03:02:02.184280 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:02.184308 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:02.184899 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:02.185305 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:02.185311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:02.185592 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:02.186339 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.186351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.187041 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.187043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:02.187375 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.187380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.187514 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.187942 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:02.187961 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:02.189607 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:02.191549 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:02.191553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:02.191559 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:02.191743 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:02.192002 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:02.192007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:02.192009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:02.192124 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:02.192611 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:02.192620 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:02.192865 (+ 245us) client_negotiation.cc:769] Sending connection context
0131 03:02:02.193016 (+ 151us) client_negotiation.cc:240] Negotiation successful
0131 03:02:02.193186 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":116,"threads_started":1}
I20260131 03:02:02.194765 7144 heartbeater.cc:344] Connected to a master server at 127.6.22.126:41235
I20260131 03:02:02.194959 7144 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:02.195350 7144 heartbeater.cc:507] Master 127.6.22.126:41235 requested a full tablet report, sending...
I20260131 03:02:02.196201 6815 ts_manager.cc:194] Registered new tserver with Master: 5e7548ab170e4bada22612cf2894212a (127.6.22.66:36101)
I20260131 03:02:02.196722 6815 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:50165
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:02.230631 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:41235
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:02.321054 7151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:02.321208 7151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:02.321254 7151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:02.323886 7151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:02.323930 7151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:02.323992 7151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:02:02.327262 7151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:41235
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:02.327998 7151 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:02.328603 7151 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:02.334095 7157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:02.334074 7156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:02.334215 7151 server_base.cc:1060] running on GCE node
W20260131 03:02:02.334074 7159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:02.334585 7151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:02.335021 7151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:02.336158 7151 hybrid_clock.cc:648] HybridClock initialized: now 1769828522336147 us; error 23 us; skew 500 ppm
Jan 31 03:02:02 dist-test-slave-j368 krb5kdc[6767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828522, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:02.338255 7151 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:02:02.339053 7151 webserver.cc:492] Webserver started at http://127.6.22.67:34247/ using document root <none> and password file <none>
I20260131 03:02:02.339565 7151 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:02.339635 7151 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:02.339815 7151 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:02.341279 7151 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "b5821794ccff4f289f35c42ef8b1ddf7"
format_stamp: "Formatted at 2026-01-31 03:02:02 on dist-test-slave-j368"
server_key: "484dbc56fdb1e737ea8768e262e88bef"
server_key_iv: "315da935bd929774eb050fcb2e53ebb5"
server_key_version: "encryptionkey@0"
I20260131 03:02:02.341676 7151 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "b5821794ccff4f289f35c42ef8b1ddf7"
format_stamp: "Formatted at 2026-01-31 03:02:02 on dist-test-slave-j368"
server_key: "484dbc56fdb1e737ea8768e262e88bef"
server_key_iv: "315da935bd929774eb050fcb2e53ebb5"
server_key_version: "encryptionkey@0"
I20260131 03:02:02.344625 7151 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260131 03:02:02.346428 7166 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:02.347240 7151 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260131 03:02:02.347348 7151 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "b5821794ccff4f289f35c42ef8b1ddf7"
format_stamp: "Formatted at 2026-01-31 03:02:02 on dist-test-slave-j368"
server_key: "484dbc56fdb1e737ea8768e262e88bef"
server_key_iv: "315da935bd929774eb050fcb2e53ebb5"
server_key_version: "encryptionkey@0"
I20260131 03:02:02.347429 7151 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:02.367177 7151 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:02.369685 7151 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:02.369863 7151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:02.370365 7151 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:02.371076 7151 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:02.371136 7151 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:02.371192 7151 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:02.371228 7151 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:02.378870 7151 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:39821
I20260131 03:02:02.378893 7279 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:39821 every 8 connection(s)
I20260131 03:02:02.379654 7151 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:02:02.384469 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 7151
I20260131 03:02:02.384570 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:02:02.384829 6233 external_mini_cluster.cc:1468] Setting key 6267967cd79bcd1dc0ad42c848c2a1c5
Jan 31 03:02:02 dist-test-slave-j368 krb5kdc[6767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828522, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:02.390405 6854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.381149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:42631 (local address 127.6.22.126:41235)
0131 03:02:02.381294 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:02.381298 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:02.381830 (+ 532us) server_negotiation.cc:408] Connection header received
0131 03:02:02.382555 (+ 725us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:02.382558 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:02.382621 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:02.382688 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:02.384078 (+ 1390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.384520 (+ 442us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.385212 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.385378 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.387343 (+ 1965us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:02.387357 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:02.387360 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:02.387385 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:02.388503 (+ 1118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:02.388991 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:02.388994 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:02.388996 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:02.389049 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:02.389365 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:02.389368 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:02.389370 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:02.389511 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:02.389614 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:02.390041 (+ 427us) server_negotiation.cc:300] Negotiation successful
0131 03:02:02.390198 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260131 03:02:02.390882 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.381353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41235 (local address 127.6.22.67:42631)
0131 03:02:02.381707 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:02.381735 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:02.382389 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:02.382825 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:02.382831 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:02.383142 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:02.383931 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.383943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.384712 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.384714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:02.385085 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.385092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.385300 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.385824 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:02.385854 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:02.387215 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:02.388617 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:02.388622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:02.388631 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:02.388878 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:02.389177 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:02.389186 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:02.389187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:02.389264 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:02.389612 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:02.389619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:02.389812 (+ 193us) client_negotiation.cc:769] Sending connection context
0131 03:02:02.389997 (+ 185us) client_negotiation.cc:240] Negotiation successful
0131 03:02:02.390228 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
I20260131 03:02:02.391745 7280 heartbeater.cc:344] Connected to a master server at 127.6.22.126:41235
I20260131 03:02:02.391943 7280 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:02.392366 7280 heartbeater.cc:507] Master 127.6.22.126:41235 requested a full tablet report, sending...
I20260131 03:02:02.393283 6815 ts_manager.cc:194] Registered new tserver with Master: b5821794ccff4f289f35c42ef8b1ddf7 (127.6.22.67:39821)
I20260131 03:02:02.393759 6815 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:42631
I20260131 03:02:02.398017 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:02:02.405476 6854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.400483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.6.22.126:41235)
0131 03:02:02.400609 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:02.400613 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:02.400792 (+ 179us) server_negotiation.cc:408] Connection header received
0131 03:02:02.400888 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:02.400890 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:02.400923 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:02.400973 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:02.401586 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.401961 (+ 375us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.402469 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.402570 (+ 101us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.403425 (+ 855us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:02.403436 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:02.403437 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:02.403452 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:02.404409 (+ 957us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:02.404740 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:02.404742 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:02.404743 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:02.404776 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:02.404935 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:02.404936 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:02.404937 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:02.405052 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:02.405131 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:02.405256 (+ 125us) server_negotiation.cc:300] Negotiation successful
0131 03:02:02.405325 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260131 03:02:02.408717 6815 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:02:02.410984 6815 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:02:02.421502 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.417880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:45751 (local address 127.0.0.1:43508)
0131 03:02:02.418279 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:02.418290 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:02.418357 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:02.419004 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:02.419007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:02.419021 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:02.419248 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.419256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.420466 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.420469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:02.421108 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.421115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.421207 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.421248 (+ 41us) client_negotiation.cc:769] Sending connection context
0131 03:02:02.421320 (+ 72us) client_negotiation.cc:240] Negotiation successful
0131 03:02:02.421391 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":98,"threads_started":1}
I20260131 03:02:02.422426 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.417650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:36101 (local address 127.0.0.1:47136)
0131 03:02:02.417939 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:02.417976 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:02.418089 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:02.418813 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:02.418845 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:02.418868 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:02.419173 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.419181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.421024 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.421028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:02.421984 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.421994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.422132 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.422149 (+ 17us) client_negotiation.cc:769] Sending connection context
0131 03:02:02.422241 (+ 92us) client_negotiation.cc:240] Negotiation successful
0131 03:02:02.422295 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260131 03:02:02.422629 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.418233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.6.22.65:45751)
0131 03:02:02.418593 (+ 360us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:02.418599 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:02.418621 (+ 22us) server_negotiation.cc:408] Connection header received
0131 03:02:02.418701 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:02.418706 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:02.418857 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:02.419006 (+ 149us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:02.419364 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.420347 (+ 983us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.421706 (+ 1359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.422250 (+ 544us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.422343 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:02.422395 (+ 52us) server_negotiation.cc:300] Negotiation successful
0131 03:02:02.422493 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":98,"threads_started":1}
I20260131 03:02:02.423424 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.418573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:39821 (local address 127.0.0.1:59964)
0131 03:02:02.418864 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:02.418878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:02.418957 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:02.419719 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:02.419722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:02.419739 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:02.419980 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.419986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.421219 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.421223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:02.421981 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.421992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.422111 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.422128 (+ 17us) client_negotiation.cc:769] Sending connection context
0131 03:02:02.422179 (+ 51us) client_negotiation.cc:240] Negotiation successful
0131 03:02:02.422272 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":75,"threads_started":1}
I20260131 03:02:02.423529 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.418695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.6.22.67:39821)
0131 03:02:02.419246 (+ 551us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:02.419250 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:02.419265 (+ 15us) server_negotiation.cc:408] Connection header received
0131 03:02:02.419344 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:02.419349 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:02.419494 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:02.419612 (+ 118us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:02.420108 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.421117 (+ 1009us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.422865 (+ 1748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.423206 (+ 341us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.423269 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:02.423342 (+ 73us) server_negotiation.cc:300] Negotiation successful
0131 03:02:02.423415 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":115,"threads_started":1}
I20260131 03:02:02.423636 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.417817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.6.22.66:36101)
0131 03:02:02.418359 (+ 542us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:02.418365 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:02.418386 (+ 21us) server_negotiation.cc:408] Connection header received
0131 03:02:02.418470 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:02.418474 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:02.418639 (+ 165us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:02.419434 (+ 795us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:02.419782 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.420917 (+ 1135us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.422759 (+ 1842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.423260 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.423360 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:02.423414 (+ 54us) server_negotiation.cc:300] Negotiation successful
0131 03:02:02.423498 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":89,"threads_started":1}
I20260131 03:02:02.425436 6942 tablet_service.cc:1505] Processing CreateTablet for tablet 9954b01a666c4474859a354755afec3d (DEFAULT_TABLE table=test-table [id=c3d9a223760c489eaa5395ceee010511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:02.425724 7214 tablet_service.cc:1505] Processing CreateTablet for tablet 9954b01a666c4474859a354755afec3d (DEFAULT_TABLE table=test-table [id=c3d9a223760c489eaa5395ceee010511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:02.426390 6942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9954b01a666c4474859a354755afec3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:02.426144 7078 tablet_service.cc:1505] Processing CreateTablet for tablet 9954b01a666c4474859a354755afec3d (DEFAULT_TABLE table=test-table [id=c3d9a223760c489eaa5395ceee010511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:02.426483 7214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9954b01a666c4474859a354755afec3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:02.426936 7078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9954b01a666c4474859a354755afec3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:02.431382 7297 tablet_bootstrap.cc:492] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7: Bootstrap starting.
I20260131 03:02:02.433120 7298 tablet_bootstrap.cc:492] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c: Bootstrap starting.
I20260131 03:02:02.433593 7297 tablet_bootstrap.cc:654] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:02.434455 7297 log.cc:826] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:02.434854 7299 tablet_bootstrap.cc:492] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a: Bootstrap starting.
I20260131 03:02:02.435127 7298 tablet_bootstrap.cc:654] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:02.435926 7298 log.cc:826] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:02.436151 7297 tablet_bootstrap.cc:492] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7: No bootstrap required, opened a new log
I20260131 03:02:02.436372 7297 ts_tablet_manager.cc:1403] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260131 03:02:02.436846 7299 tablet_bootstrap.cc:654] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:02.437492 7299 log.cc:826] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:02.437729 7298 tablet_bootstrap.cc:492] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c: No bootstrap required, opened a new log
I20260131 03:02:02.437903 7298 ts_tablet_manager.cc:1403] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260131 03:02:02.438993 7299 tablet_bootstrap.cc:492] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a: No bootstrap required, opened a new log
I20260131 03:02:02.439185 7299 ts_tablet_manager.cc:1403] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260131 03:02:02.440202 7297 raft_consensus.cc:359] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7548ab170e4bada22612cf2894212a" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36101 } } peers { permanent_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45751 } } peers { permanent_uuid: "b5821794ccff4f289f35c42ef8b1ddf7" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39821 } }
I20260131 03:02:02.440476 7297 raft_consensus.cc:385] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:02.440539 7297 raft_consensus.cc:740] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5821794ccff4f289f35c42ef8b1ddf7, State: Initialized, Role: FOLLOWER
I20260131 03:02:02.441103 7297 consensus_queue.cc:260] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7548ab170e4bada22612cf2894212a" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36101 } } peers { permanent_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45751 } } peers { permanent_uuid: "b5821794ccff4f289f35c42ef8b1ddf7" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39821 } }
I20260131 03:02:02.441759 7280 heartbeater.cc:499] Master 127.6.22.126:41235 was elected leader, sending a full tablet report...
I20260131 03:02:02.441766 7298 raft_consensus.cc:359] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7548ab170e4bada22612cf2894212a" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36101 } } peers { permanent_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45751 } } peers { permanent_uuid: "b5821794ccff4f289f35c42ef8b1ddf7" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39821 } }
I20260131 03:02:02.441969 7298 raft_consensus.cc:385] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:02.442034 7298 raft_consensus.cc:740] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d917c6a566a4c6d895ab8477f2c4a9c, State: Initialized, Role: FOLLOWER
I20260131 03:02:02.442058 7297 ts_tablet_manager.cc:1434] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260131 03:02:02.442468 7298 consensus_queue.cc:260] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7548ab170e4bada22612cf2894212a" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36101 } } peers { permanent_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45751 } } peers { permanent_uuid: "b5821794ccff4f289f35c42ef8b1ddf7" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39821 } }
I20260131 03:02:02.442502 7299 raft_consensus.cc:359] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7548ab170e4bada22612cf2894212a" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36101 } } peers { permanent_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45751 } } peers { permanent_uuid: "b5821794ccff4f289f35c42ef8b1ddf7" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39821 } }
I20260131 03:02:02.442695 7299 raft_consensus.cc:385] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:02.442761 7299 raft_consensus.cc:740] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7548ab170e4bada22612cf2894212a, State: Initialized, Role: FOLLOWER
I20260131 03:02:02.443252 7008 heartbeater.cc:499] Master 127.6.22.126:41235 was elected leader, sending a full tablet report...
I20260131 03:02:02.443214 7299 consensus_queue.cc:260] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7548ab170e4bada22612cf2894212a" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36101 } } peers { permanent_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45751 } } peers { permanent_uuid: "b5821794ccff4f289f35c42ef8b1ddf7" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39821 } }
I20260131 03:02:02.443357 7298 ts_tablet_manager.cc:1434] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260131 03:02:02.444013 7299 ts_tablet_manager.cc:1434] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260131 03:02:02.444308 7144 heartbeater.cc:499] Master 127.6.22.126:41235 was elected leader, sending a full tablet report...
W20260131 03:02:02.485435 7009 tablet.cc:2378] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:02:02.630520 7281 tablet.cc:2378] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:02:02.636983 7304 raft_consensus.cc:493] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:02:02.637151 7304 raft_consensus.cc:515] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7548ab170e4bada22612cf2894212a" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36101 } } peers { permanent_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45751 } } peers { permanent_uuid: "b5821794ccff4f289f35c42ef8b1ddf7" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39821 } }
I20260131 03:02:02.638180 7304 leader_election.cc:290] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e7548ab170e4bada22612cf2894212a (127.6.22.66:36101), b5821794ccff4f289f35c42ef8b1ddf7 (127.6.22.67:39821)
I20260131 03:02:02.641752 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.638397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:39821 (local address 127.6.22.65:34975)
0131 03:02:02.638724 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:02.638739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:02.638852 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:02.639107 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:02.639110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:02.639143 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:02.639484 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.639492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.640500 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.640504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:02.641330 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.641339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.641447 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.641464 (+ 17us) client_negotiation.cc:769] Sending connection context
0131 03:02:02.641511 (+ 47us) client_negotiation.cc:240] Negotiation successful
0131 03:02:02.641562 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"spinlock_wait_cycles":11136,"thread_start_us":93,"threads_started":1}
I20260131 03:02:02.641753 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.638397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:36101 (local address 127.6.22.65:46411)
0131 03:02:02.638770 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:02.638781 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:02.638856 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:02.639110 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:02.639112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:02.639143 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:02.639486 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.639492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.640500 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.640504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:02.641330 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:02.641339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.641445 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.641465 (+ 20us) client_negotiation.cc:769] Sending connection context
0131 03:02:02.641511 (+ 46us) client_negotiation.cc:240] Negotiation successful
0131 03:02:02.641562 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":9,"spinlock_wait_cycles":21376,"thread_start_us":82,"threads_started":1}
I20260131 03:02:02.642091 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.638507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:46411 (local address 127.6.22.66:36101)
0131 03:02:02.638636 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:02.638641 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:02.638845 (+ 204us) server_negotiation.cc:408] Connection header received
0131 03:02:02.638948 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:02.638951 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:02.639001 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:02.639086 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:02.639607 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.640396 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.641443 (+ 1047us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.641866 (+ 423us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.641898 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:02.641955 (+ 57us) server_negotiation.cc:300] Negotiation successful
0131 03:02:02.641998 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260131 03:02:02.642091 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.638573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:34975 (local address 127.6.22.67:39821)
0131 03:02:02.638676 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:02.638679 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:02.638809 (+ 130us) server_negotiation.cc:408] Connection header received
0131 03:02:02.638948 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:02.638951 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:02.639001 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:02.639086 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:02.639607 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.640396 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.641443 (+ 1047us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.641866 (+ 423us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.641898 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:02.641942 (+ 44us) server_negotiation.cc:300] Negotiation successful
0131 03:02:02.641992 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260131 03:02:02.642581 7234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9954b01a666c4474859a354755afec3d" candidate_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5821794ccff4f289f35c42ef8b1ddf7" is_pre_election: true
I20260131 03:02:02.642588 7098 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9954b01a666c4474859a354755afec3d" candidate_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7548ab170e4bada22612cf2894212a" is_pre_election: true
I20260131 03:02:02.642798 7098 raft_consensus.cc:2468] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d917c6a566a4c6d895ab8477f2c4a9c in term 0.
I20260131 03:02:02.642808 7234 raft_consensus.cc:2468] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d917c6a566a4c6d895ab8477f2c4a9c in term 0.
I20260131 03:02:02.643213 6898 leader_election.cc:304] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d917c6a566a4c6d895ab8477f2c4a9c, 5e7548ab170e4bada22612cf2894212a; no voters:
I20260131 03:02:02.643421 7304 raft_consensus.cc:2804] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:02:02.643481 7304 raft_consensus.cc:493] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:02:02.643525 7304 raft_consensus.cc:3060] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:02.644345 7304 raft_consensus.cc:515] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7548ab170e4bada22612cf2894212a" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36101 } } peers { permanent_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45751 } } peers { permanent_uuid: "b5821794ccff4f289f35c42ef8b1ddf7" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39821 } }
I20260131 03:02:02.644644 7304 leader_election.cc:290] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [CANDIDATE]: Term 1 election: Requested vote from peers 5e7548ab170e4bada22612cf2894212a (127.6.22.66:36101), b5821794ccff4f289f35c42ef8b1ddf7 (127.6.22.67:39821)
I20260131 03:02:02.644958 7098 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9954b01a666c4474859a354755afec3d" candidate_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7548ab170e4bada22612cf2894212a"
I20260131 03:02:02.645092 7098 raft_consensus.cc:3060] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:02.645855 7098 raft_consensus.cc:2468] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d917c6a566a4c6d895ab8477f2c4a9c in term 1.
I20260131 03:02:02.646086 7234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9954b01a666c4474859a354755afec3d" candidate_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5821794ccff4f289f35c42ef8b1ddf7"
I20260131 03:02:02.646148 6898 leader_election.cc:304] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d917c6a566a4c6d895ab8477f2c4a9c, 5e7548ab170e4bada22612cf2894212a; no voters:
I20260131 03:02:02.646191 7234 raft_consensus.cc:3060] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:02.646334 7304 raft_consensus.cc:2804] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:02.646519 7304 raft_consensus.cc:697] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [term 1 LEADER]: Becoming Leader. State: Replica: 4d917c6a566a4c6d895ab8477f2c4a9c, State: Running, Role: LEADER
I20260131 03:02:02.646811 7304 consensus_queue.cc:237] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7548ab170e4bada22612cf2894212a" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36101 } } peers { permanent_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45751 } } peers { permanent_uuid: "b5821794ccff4f289f35c42ef8b1ddf7" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39821 } }
I20260131 03:02:02.647225 7234 raft_consensus.cc:2468] T 9954b01a666c4474859a354755afec3d P b5821794ccff4f289f35c42ef8b1ddf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d917c6a566a4c6d895ab8477f2c4a9c in term 1.
I20260131 03:02:02.649673 6814 catalog_manager.cc:5654] T 9954b01a666c4474859a354755afec3d P 4d917c6a566a4c6d895ab8477f2c4a9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d917c6a566a4c6d895ab8477f2c4a9c (127.6.22.65). New cstate: current_term: 1 leader_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7548ab170e4bada22612cf2894212a" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d917c6a566a4c6d895ab8477f2c4a9c" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5821794ccff4f289f35c42ef8b1ddf7" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39821 } health_report { overall_health: UNKNOWN } } }
W20260131 03:02:02.683204 7145 tablet.cc:2378] T 9954b01a666c4474859a354755afec3d P 5e7548ab170e4bada22612cf2894212a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jan 31 03:02:02 dist-test-slave-j368 krb5kdc[6767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.65@KRBTEST.COM
I20260131 03:02:02.712100 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.705082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.6.22.65:45751)
0131 03:02:02.705214 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:02.705219 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:02.705355 (+ 136us) server_negotiation.cc:408] Connection header received
0131 03:02:02.705494 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:02.705497 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:02.705544 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:02.705619 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:02.706361 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.706813 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.707432 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.707685 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.709561 (+ 1876us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:02.709583 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:02.709594 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:02.709618 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:02.711028 (+ 1410us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:02.711343 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:02.711348 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:02.711349 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:02.711397 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:02.711547 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:02.711549 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:02.711550 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:02.711684 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:02.711828 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:02.711912 (+ 84us) server_negotiation.cc:300] Negotiation successful
0131 03:02:02.711996 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Jan 31 03:02:02 dist-test-slave-j368 krb5kdc[6767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 31 03:02:02 dist-test-slave-j368 krb5kdc[6767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.22.65@KRBTEST.COM
I20260131 03:02:02.732939 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.726603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.6.22.65:45751)
0131 03:02:02.726726 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:02.726729 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:02.726826 (+ 97us) server_negotiation.cc:408] Connection header received
0131 03:02:02.726954 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:02.726957 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:02.726997 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:02.727069 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:02.727684 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.728080 (+ 396us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.728625 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.728819 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.730626 (+ 1807us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:02.730646 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:02.730648 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:02.730667 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:02.731642 (+ 975us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:02.731964 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:02.731967 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:02.731971 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:02.732010 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:02.732223 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:02.732227 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:02.732229 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:02.732334 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:02.732465 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:02.732689 (+ 224us) server_negotiation.cc:300] Negotiation successful
0131 03:02:02.732778 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jan 31 03:02:02 dist-test-slave-j368 krb5kdc[6767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 31 03:02:02 dist-test-slave-j368 krb5kdc[6767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.65@KRBTEST.COM
I20260131 03:02:02.753329 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:02.746911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.6.22.65:45751)
0131 03:02:02.747018 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:02.747023 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:02.747143 (+ 120us) server_negotiation.cc:408] Connection header received
0131 03:02:02.747227 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:02.747229 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:02.747260 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:02.747315 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:02.747884 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.748251 (+ 367us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:02.748786 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:02.748888 (+ 102us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:02.750714 (+ 1826us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:02.750725 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:02.750727 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:02.750745 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:02.752013 (+ 1268us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:02.752348 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:02.752351 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:02.752353 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:02.752396 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:02.752607 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:02.752610 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:02.752612 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:02.752778 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:02.752895 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:02.753097 (+ 202us) server_negotiation.cc:300] Negotiation successful
0131 03:02:02.753195 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260131 03:02:02.754451 6922 tablet_service.cc:3012] Rejecting scan request for tablet 9954b01a666c4474859a354755afec3d: Uninitialized: clean time has not yet been initialized
I20260131 03:02:02.755893 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 6879
I20260131 03:02:02.761245 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 7015
I20260131 03:02:02.766170 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 7151
I20260131 03:02:02.771153 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 6783
2026-01-31T03:02:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3285 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:02:02 dist-test-slave-j368 krb5kdc[7322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:02:02 dist-test-slave-j368 krb5kdc[7322](info): set up 2 sockets
Jan 31 03:02:02 dist-test-slave-j368 krb5kdc[7322](info): commencing operation
krb5kdc: starting...
W20260131 03:02:04.645208 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.852s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260131 03:02:04.877372 6233 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.232s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260131 03:02:05.001546 6233 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.124s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260131 03:02:05.179044 6233 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.177s user 0.002s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:02:05 dist-test-slave-j368 krb5kdc[7322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:02:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:05.250061 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:38373
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:05.366106 7338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:05.366305 7338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:05.366389 7338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:05.369055 7338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:05.369124 7338 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260131 03:02:05.369158 7338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:05.369208 7338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:05.369257 7338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:05.369292 7338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:05.372814 7338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:38373
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:05.373698 7338 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:05.374413 7338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:05.379254 7343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:05.379325 7344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:05.379257 7346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:05.379520 7338 server_base.cc:1060] running on GCE node
I20260131 03:02:05.379935 7338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:05.380787 7338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:05.381924 7338 hybrid_clock.cc:648] HybridClock initialized: now 1769828525381912 us; error 35 us; skew 500 ppm
Jan 31 03:02:05 dist-test-slave-j368 krb5kdc[7322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:05.383944 7338 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:02:05.384864 7338 webserver.cc:492] Webserver started at http://127.6.22.126:41549/ using document root <none> and password file <none>
I20260131 03:02:05.385314 7338 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:05.385378 7338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:05.385555 7338 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:05.387033 7338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "77cb500bb87a4c94b55816101052b3e8"
format_stamp: "Formatted at 2026-01-31 03:02:05 on dist-test-slave-j368"
server_key: "20caa9d570a74af587165bc2fdadefee"
server_key_iv: "57dee98bc47e62a3dec880bb435cd17d"
server_key_version: "encryptionkey@0"
I20260131 03:02:05.387506 7338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "77cb500bb87a4c94b55816101052b3e8"
format_stamp: "Formatted at 2026-01-31 03:02:05 on dist-test-slave-j368"
server_key: "20caa9d570a74af587165bc2fdadefee"
server_key_iv: "57dee98bc47e62a3dec880bb435cd17d"
server_key_version: "encryptionkey@0"
I20260131 03:02:05.390512 7338 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260131 03:02:05.392277 7353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:05.393203 7338 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:05.393337 7338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "77cb500bb87a4c94b55816101052b3e8"
format_stamp: "Formatted at 2026-01-31 03:02:05 on dist-test-slave-j368"
server_key: "20caa9d570a74af587165bc2fdadefee"
server_key_iv: "57dee98bc47e62a3dec880bb435cd17d"
server_key_version: "encryptionkey@0"
I20260131 03:02:05.393430 7338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:05.415367 7338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:05.419631 7338 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:05.419857 7338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:05.428575 7338 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:38373
I20260131 03:02:05.428838 7415 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:38373 every 8 connection(s)
I20260131 03:02:05.429791 7338 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:05.433609 7416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:05.437110 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 7338
I20260131 03:02:05.437234 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:05.437547 6233 external_mini_cluster.cc:1468] Setting key 0ae083ff5a8d60dfad3c71e8d787c5c4
I20260131 03:02:05.440227 7416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8: Bootstrap starting.
I20260131 03:02:05.442561 7416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:05.443430 7416 log.cc:826] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:05.445441 7416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8: No bootstrap required, opened a new log
Jan 31 03:02:05 dist-test-slave-j368 krb5kdc[7322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:05.449004 7416 raft_consensus.cc:359] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cb500bb87a4c94b55816101052b3e8" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38373 } }
I20260131 03:02:05.449245 7416 raft_consensus.cc:385] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:05.449328 7416 raft_consensus.cc:740] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cb500bb87a4c94b55816101052b3e8, State: Initialized, Role: FOLLOWER
I20260131 03:02:05.449841 7416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cb500bb87a4c94b55816101052b3e8" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38373 } }
I20260131 03:02:05.449993 7416 raft_consensus.cc:399] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:05.450079 7416 raft_consensus.cc:493] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:05.450176 7416 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:05.451150 7416 raft_consensus.cc:515] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cb500bb87a4c94b55816101052b3e8" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38373 } }
I20260131 03:02:05.451376 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:05.439663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.6.22.126:38373)
0131 03:02:05.440174 (+ 511us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:05.440184 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:05.440223 (+ 39us) server_negotiation.cc:408] Connection header received
0131 03:02:05.440923 (+ 700us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:05.440943 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:05.441269 (+ 326us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:05.441733 (+ 464us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:05.442695 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.443634 (+ 939us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:05.444315 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.444609 (+ 294us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:05.446954 (+ 2345us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:05.446973 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:05.446983 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:05.447018 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:05.448789 (+ 1771us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:05.449203 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:05.449216 (+ 13us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:05.449233 (+ 17us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:05.449337 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:05.449599 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:05.449602 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:05.449604 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:05.449939 (+ 335us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:05.450200 (+ 261us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:05.450448 (+ 248us) server_negotiation.cc:300] Negotiation successful
0131 03:02:05.450729 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":169,"threads_started":1}
I20260131 03:02:05.451524 7416 leader_election.cc:304] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77cb500bb87a4c94b55816101052b3e8; no voters:
I20260131 03:02:05.451907 7416 leader_election.cc:290] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:05.451983 7421 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:05.452152 7421 raft_consensus.cc:697] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [term 1 LEADER]: Becoming Leader. State: Replica: 77cb500bb87a4c94b55816101052b3e8, State: Running, Role: LEADER
I20260131 03:02:05.453025 7416 sys_catalog.cc:565] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:05.453420 7421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cb500bb87a4c94b55816101052b3e8" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38373 } }
I20260131 03:02:05.454977 7423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77cb500bb87a4c94b55816101052b3e8. Latest consensus state: current_term: 1 leader_uuid: "77cb500bb87a4c94b55816101052b3e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cb500bb87a4c94b55816101052b3e8" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38373 } } }
I20260131 03:02:05.455114 7422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77cb500bb87a4c94b55816101052b3e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cb500bb87a4c94b55816101052b3e8" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38373 } } }
I20260131 03:02:05.455454 7422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:05.455451 7423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8 [sys.catalog]: This master's current role is: LEADER
W20260131 03:02:05.457366 7435 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:02:05.457443 7435 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:02:05.457512 7436 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:05.458256 7436 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:05.462998 7436 catalog_manager.cc:1357] Generated new cluster ID: a1a2444e378141ffb439b06d0e54da73
I20260131 03:02:05.463068 7436 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:05.472134 7436 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:05.473070 7436 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:05.487736 7436 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8: Generated new TSK 0
I20260131 03:02:05.488247 7436 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:05.539047 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:38373
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:05.650836 7444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:05.651098 7444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:05.651160 7444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:05.654799 7444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:05.654968 7444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:05.655078 7444 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:02:05.655154 7444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:02:05.660207 7444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.22.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:05.661407 7444 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:05.662310 7444 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:05.668996 7449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:05.669111 7450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:05.668998 7452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:05.669299 7444 server_base.cc:1060] running on GCE node
I20260131 03:02:05.669687 7444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:05.676757 7444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:05.678022 7444 hybrid_clock.cc:648] HybridClock initialized: now 1769828525678012 us; error 53 us; skew 500 ppm
Jan 31 03:02:05 dist-test-slave-j368 krb5kdc[7322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:05.681385 7444 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:02:05.682826 7444 webserver.cc:492] Webserver started at http://127.6.22.65:36599/ using document root <none> and password file <none>
I20260131 03:02:05.683507 7444 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:05.683562 7444 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:05.683754 7444 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:05.685948 7444 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "b23a177155a34f22a836aa46035ad1f1"
format_stamp: "Formatted at 2026-01-31 03:02:05 on dist-test-slave-j368"
server_key: "e52733f4bebc1e794aef005f0f0261cd"
server_key_iv: "4b157a656fa536c9d9af5204726563b8"
server_key_version: "encryptionkey@0"
I20260131 03:02:05.686543 7444 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "b23a177155a34f22a836aa46035ad1f1"
format_stamp: "Formatted at 2026-01-31 03:02:05 on dist-test-slave-j368"
server_key: "e52733f4bebc1e794aef005f0f0261cd"
server_key_iv: "4b157a656fa536c9d9af5204726563b8"
server_key_version: "encryptionkey@0"
I20260131 03:02:05.691092 7444 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260131 03:02:05.694144 7459 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:05.695554 7444 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:02:05.695726 7444 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "b23a177155a34f22a836aa46035ad1f1"
format_stamp: "Formatted at 2026-01-31 03:02:05 on dist-test-slave-j368"
server_key: "e52733f4bebc1e794aef005f0f0261cd"
server_key_iv: "4b157a656fa536c9d9af5204726563b8"
server_key_version: "encryptionkey@0"
I20260131 03:02:05.695851 7444 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:05.711381 7444 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:05.714453 7444 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:05.714663 7444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:05.716428 7444 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:05.716495 7444 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:05.716559 7444 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:05.716614 7444 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:02:05 dist-test-slave-j368 krb5kdc[7322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:05.727911 7444 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:40273
I20260131 03:02:05.728106 7574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:40273 every 8 connection(s)
I20260131 03:02:05.729091 7444 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:02:05.730763 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:05.717153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:50035 (local address 127.6.22.126:38373)
0131 03:02:05.717320 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:05.717327 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:05.718567 (+ 1240us) server_negotiation.cc:408] Connection header received
0131 03:02:05.719805 (+ 1238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:05.719813 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:05.719887 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:05.719954 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:05.722127 (+ 2173us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.722705 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:05.723701 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.723929 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:05.726768 (+ 2839us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:05.726801 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:05.726804 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:05.726834 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:05.728568 (+ 1734us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:05.729160 (+ 592us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:05.729166 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:05.729168 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:05.729244 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:05.729583 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:05.729587 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:05.729589 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:05.729770 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:05.729888 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:05.730426 (+ 538us) server_negotiation.cc:300] Negotiation successful
0131 03:02:05.730579 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260131 03:02:05.731429 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:05.717487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38373 (local address 127.6.22.65:50035)
0131 03:02:05.718386 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:05.718434 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:05.719531 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:05.720418 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:05.720429 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:05.721095 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:05.721932 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:05.721954 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.722853 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:05.722859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:05.723546 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:05.723561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.723805 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:05.724503 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:05.724534 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:05.726599 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:05.728758 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:05.728764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:05.728776 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:05.729043 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:05.729364 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:05.729369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:05.729372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:05.729474 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:05.729888 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:05.729898 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:05.730160 (+ 262us) client_negotiation.cc:769] Sending connection context
0131 03:02:05.730373 (+ 213us) client_negotiation.cc:240] Negotiation successful
0131 03:02:05.730646 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":527,"threads_started":1}
I20260131 03:02:05.732836 7575 heartbeater.cc:344] Connected to a master server at 127.6.22.126:38373
I20260131 03:02:05.733031 7575 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:05.733635 7575 heartbeater.cc:507] Master 127.6.22.126:38373 requested a full tablet report, sending...
I20260131 03:02:05.735184 7369 ts_manager.cc:194] Registered new tserver with Master: b23a177155a34f22a836aa46035ad1f1 (127.6.22.65:40273)
I20260131 03:02:05.736459 7369 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:50035
I20260131 03:02:05.737514 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 7444
I20260131 03:02:05.737635 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:02:05.738371 6233 external_mini_cluster.cc:1468] Setting key cf0d19de9496345360c52a7525284be7
I20260131 03:02:05.741837 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:05.734845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38373 (local address 127.6.22.65:54439)
0131 03:02:05.735192 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:05.735203 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:05.735313 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:05.735575 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:05.735578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:05.735754 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:05.735967 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:05.735973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.736755 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:05.736760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:05.737308 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:05.737321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.737499 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:05.738116 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:05.738136 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:05.738553 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:05.740274 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:05.740280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:05.740285 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:05.740529 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:05.740821 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:05.740828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:05.740833 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:05.740907 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:05.741237 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:05.741243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:05.741356 (+ 113us) client_negotiation.cc:769] Sending connection context
0131 03:02:05.741489 (+ 133us) client_negotiation.cc:240] Negotiation successful
0131 03:02:05.741653 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":101,"threads_started":1}
I20260131 03:02:05.741896 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:05.734913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:54439 (local address 127.6.22.126:38373)
0131 03:02:05.735072 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:05.735075 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:05.735283 (+ 208us) server_negotiation.cc:408] Connection header received
0131 03:02:05.735427 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:05.735432 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:05.735476 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:05.735558 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:05.736089 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.736577 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:05.737496 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.737678 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:05.738690 (+ 1012us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:05.738709 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:05.738712 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:05.738737 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:05.740157 (+ 1420us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:05.740634 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:05.740637 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:05.740638 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:05.740729 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:05.740998 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:05.741000 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:05.741001 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:05.741138 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:05.741210 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:05.741565 (+ 355us) server_negotiation.cc:300] Negotiation successful
0131 03:02:05.741698 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:05.796357 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:38373
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:05.914898 7586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:05.915283 7586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:05.915390 7586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:05.919497 7586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:05.919675 7586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:05.919800 7586 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:02:05.919909 7586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:02:05.925467 7586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.22.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:05.926717 7586 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:05.927711 7586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:05.935263 7592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:05.935253 7591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:05.935253 7594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:05.935657 7586 server_base.cc:1060] running on GCE node
I20260131 03:02:05.936187 7586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:05.937000 7586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:05.938244 7586 hybrid_clock.cc:648] HybridClock initialized: now 1769828525938213 us; error 52 us; skew 500 ppm
Jan 31 03:02:05 dist-test-slave-j368 krb5kdc[7322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:05.941634 7586 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:02:05.942797 7586 webserver.cc:492] Webserver started at http://127.6.22.66:39213/ using document root <none> and password file <none>
I20260131 03:02:05.943397 7586 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:05.943444 7586 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:05.943610 7586 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:05.945525 7586 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "362f6a72983a4499b326b3e277ba7368"
format_stamp: "Formatted at 2026-01-31 03:02:05 on dist-test-slave-j368"
server_key: "948fc9144bfe50f1687f5903bbc838b8"
server_key_iv: "f589cce10233448652e67a52ef2d02f1"
server_key_version: "encryptionkey@0"
I20260131 03:02:05.945969 7586 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "362f6a72983a4499b326b3e277ba7368"
format_stamp: "Formatted at 2026-01-31 03:02:05 on dist-test-slave-j368"
server_key: "948fc9144bfe50f1687f5903bbc838b8"
server_key_iv: "f589cce10233448652e67a52ef2d02f1"
server_key_version: "encryptionkey@0"
I20260131 03:02:05.949630 7586 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260131 03:02:05.952142 7601 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:05.953387 7586 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260131 03:02:05.953514 7586 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "362f6a72983a4499b326b3e277ba7368"
format_stamp: "Formatted at 2026-01-31 03:02:05 on dist-test-slave-j368"
server_key: "948fc9144bfe50f1687f5903bbc838b8"
server_key_iv: "f589cce10233448652e67a52ef2d02f1"
server_key_version: "encryptionkey@0"
I20260131 03:02:05.953707 7586 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:05.968334 7586 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:05.971472 7586 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:05.971637 7586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:05.973328 7586 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:05.973379 7586 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:05.973418 7586 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:05.973490 7586 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:02:05 dist-test-slave-j368 krb5kdc[7322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:05.987026 7586 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:43507
I20260131 03:02:05.987080 7716 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:43507 every 8 connection(s)
I20260131 03:02:05.988309 7586 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:02:05.988817 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:05.974029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:39017 (local address 127.6.22.126:38373)
0131 03:02:05.974183 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:05.974189 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:05.974946 (+ 757us) server_negotiation.cc:408] Connection header received
0131 03:02:05.975977 (+ 1031us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:05.975981 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:05.976047 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:05.976172 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:05.978642 (+ 2470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.979537 (+ 895us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:05.980690 (+ 1153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.980928 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:05.984229 (+ 3301us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:05.984261 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:05.984268 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:05.984311 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:05.986376 (+ 2065us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:05.987203 (+ 827us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:05.987207 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:05.987208 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:05.987272 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:05.987621 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:05.987624 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:05.987627 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:05.987807 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:05.987917 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:05.988494 (+ 577us) server_negotiation.cc:300] Negotiation successful
0131 03:02:05.988622 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260131 03:02:05.989557 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:05.974323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38373 (local address 127.6.22.66:39017)
0131 03:02:05.974767 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:05.974808 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:05.975625 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:05.976497 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:05.976513 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:05.977334 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:05.978350 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:05.978397 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.979732 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:05.979736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:05.980497 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:05.980509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.980850 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:05.981767 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:05.981828 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:05.983989 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:05.986639 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:05.986648 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:05.986661 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:05.987061 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:05.987381 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:05.987386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:05.987393 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:05.987478 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:05.987917 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:05.987926 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:05.988207 (+ 281us) client_negotiation.cc:769] Sending connection context
0131 03:02:05.988426 (+ 219us) client_negotiation.cc:240] Negotiation successful
0131 03:02:05.988745 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":140,"threads_started":1}
I20260131 03:02:05.991096 7717 heartbeater.cc:344] Connected to a master server at 127.6.22.126:38373
I20260131 03:02:05.991330 7717 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:05.992000 7717 heartbeater.cc:507] Master 127.6.22.126:38373 requested a full tablet report, sending...
I20260131 03:02:05.993355 7370 ts_manager.cc:194] Registered new tserver with Master: 362f6a72983a4499b326b3e277ba7368 (127.6.22.66:43507)
I20260131 03:02:05.993590 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 7586
I20260131 03:02:05.993713 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:02:05.993997 6233 external_mini_cluster.cc:1468] Setting key bea5e33e61d47adb4255732991e21292
I20260131 03:02:05.994307 7370 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:39017
I20260131 03:02:06.001791 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:05.994495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:55697 (local address 127.6.22.126:38373)
0131 03:02:05.994660 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:05.994676 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:05.994694 (+ 18us) server_negotiation.cc:408] Connection header received
0131 03:02:05.994734 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:05.994737 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:05.994813 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:05.994922 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:05.995831 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.996585 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:05.997341 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.997578 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:05.998513 (+ 935us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:05.998532 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:05.998537 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:05.998567 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:06.000103 (+ 1536us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.000534 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.000538 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.000541 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.000595 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.000930 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.000933 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.000935 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.001136 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:06.001248 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.001501 (+ 253us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.001615 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260131 03:02:06.001808 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:05.993718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38373 (local address 127.6.22.66:55697)
0131 03:02:05.994109 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:05.994134 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:05.994245 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:05.995029 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:05.995033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:05.995379 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:05.995688 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:05.995697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.996749 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:05.996752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:05.997222 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:05.997230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:05.997339 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:05.998050 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:05.998064 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:05.998365 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:06.000228 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:06.000233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:06.000236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:06.000421 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:06.000749 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:06.000752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:06.000754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:06.000802 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:06.001237 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:06.001241 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:06.001339 (+ 98us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.001451 (+ 112us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.001602 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":149,"threads_started":1}
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:06.045076 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:38373
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:06.161317 7728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:06.161559 7728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:06.161674 7728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:06.165087 7728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:06.165167 7728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:06.165225 7728 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:02:06.165314 7728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:02:06.170362 7728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.22.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:06.171648 7728 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:06.172529 7728 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:06.180053 7736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:06.180022 7733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:06.180327 7728 server_base.cc:1060] running on GCE node
W20260131 03:02:06.180022 7734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:06.180945 7728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:06.181644 7728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:06.182808 7728 hybrid_clock.cc:648] HybridClock initialized: now 1769828526182785 us; error 46 us; skew 500 ppm
Jan 31 03:02:06 dist-test-slave-j368 krb5kdc[7322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828526, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:06.185854 7728 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:02:06.186986 7728 webserver.cc:492] Webserver started at http://127.6.22.67:41785/ using document root <none> and password file <none>
I20260131 03:02:06.187575 7728 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:06.187650 7728 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:06.187866 7728 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:06.189735 7728 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "c759e57501c44996a27cba6493bc2903"
format_stamp: "Formatted at 2026-01-31 03:02:06 on dist-test-slave-j368"
server_key: "ea2d7783bf489b21aa6deb0a6d12c80f"
server_key_iv: "3cb56a6b86679b0a775188f29896d7e7"
server_key_version: "encryptionkey@0"
I20260131 03:02:06.190243 7728 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "c759e57501c44996a27cba6493bc2903"
format_stamp: "Formatted at 2026-01-31 03:02:06 on dist-test-slave-j368"
server_key: "ea2d7783bf489b21aa6deb0a6d12c80f"
server_key_iv: "3cb56a6b86679b0a775188f29896d7e7"
server_key_version: "encryptionkey@0"
I20260131 03:02:06.195436 7728 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260131 03:02:06.198289 7743 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:06.199681 7728 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260131 03:02:06.199834 7728 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "c759e57501c44996a27cba6493bc2903"
format_stamp: "Formatted at 2026-01-31 03:02:06 on dist-test-slave-j368"
server_key: "ea2d7783bf489b21aa6deb0a6d12c80f"
server_key_iv: "3cb56a6b86679b0a775188f29896d7e7"
server_key_version: "encryptionkey@0"
I20260131 03:02:06.199976 7728 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:06.214790 7728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:06.218200 7728 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:06.218420 7728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:06.220118 7728 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:06.220198 7728 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:06.220273 7728 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:06.220324 7728 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:02:06 dist-test-slave-j368 krb5kdc[7322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828526, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:06.232707 7728 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:39139
I20260131 03:02:06.232791 7858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:39139 every 8 connection(s)
I20260131 03:02:06.233757 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.221324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:47465 (local address 127.6.22.126:38373)
0131 03:02:06.221538 (+ 214us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.221546 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.222444 (+ 898us) server_negotiation.cc:408] Connection header received
0131 03:02:06.223498 (+ 1054us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.223504 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.223592 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.223702 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:06.225337 (+ 1635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.225858 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.226516 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.226693 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.229153 (+ 2460us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:06.229186 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:06.229191 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:06.229226 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:06.231140 (+ 1914us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.231829 (+ 689us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.231839 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.231841 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.231895 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.232275 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.232283 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.232288 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.232557 (+ 269us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:06.232736 (+ 179us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.233423 (+ 687us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.233569 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260131 03:02:06.234092 7728 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:02:06.234669 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.221555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38373 (local address 127.6.22.67:47465)
0131 03:02:06.222152 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.222195 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.223194 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.224036 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.224051 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.224525 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:06.225183 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.225197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.225979 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.225983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.226400 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.226407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.226638 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.227271 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:06.227312 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:06.228993 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:06.231324 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:06.231331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:06.231349 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:06.231685 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:06.232000 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:06.232008 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:06.232010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:06.232111 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:06.232730 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:06.232738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:06.233110 (+ 372us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.233340 (+ 230us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.233702 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":153,"threads_started":1}
I20260131 03:02:06.236413 7859 heartbeater.cc:344] Connected to a master server at 127.6.22.126:38373
I20260131 03:02:06.236609 7859 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:06.237381 7859 heartbeater.cc:507] Master 127.6.22.126:38373 requested a full tablet report, sending...
I20260131 03:02:06.238735 7369 ts_manager.cc:194] Registered new tserver with Master: c759e57501c44996a27cba6493bc2903 (127.6.22.67:39139)
I20260131 03:02:06.239621 7369 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:47465
I20260131 03:02:06.242008 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 7728
I20260131 03:02:06.242144 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:02:06.242421 6233 external_mini_cluster.cc:1468] Setting key c0075da99562b10b8047c1204738e225
I20260131 03:02:06.244613 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:02:06.246685 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.239257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:49305 (local address 127.6.22.126:38373)
0131 03:02:06.240002 (+ 745us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.240009 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.240021 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:02:06.240054 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.240057 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.240106 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.240186 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:06.240915 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.241383 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.242207 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.242407 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.243492 (+ 1085us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:06.243508 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:06.243512 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:06.243539 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:06.245144 (+ 1605us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.245591 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.245593 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.245595 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.245633 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.245869 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.245871 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.245872 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.246005 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:06.246078 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.246370 (+ 292us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.246459 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647}
I20260131 03:02:06.246732 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.239116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38373 (local address 127.6.22.67:49305)
0131 03:02:06.239524 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.239542 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.239692 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.240223 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.240226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.240469 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:06.240787 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.240797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.241497 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.241501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.242015 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.242024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.242228 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.242947 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:06.242962 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:06.243382 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:06.245250 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:06.245255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:06.245257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:06.245486 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:06.245714 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:06.245720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:06.245725 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:06.245787 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:06.246099 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:06.246105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:06.246208 (+ 103us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.246323 (+ 115us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.246475 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":154,"threads_started":1}
Jan 31 03:02:06 dist-test-slave-j368 krb5kdc[7322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 31 03:02:06 dist-test-slave-j368 krb5kdc[7322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:06.272913 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.264186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.6.22.126:38373)
0131 03:02:06.264440 (+ 254us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.264444 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.264459 (+ 15us) server_negotiation.cc:408] Connection header received
0131 03:02:06.264506 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.264509 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.264558 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.264637 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:06.265501 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.266010 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.266902 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.267045 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.269658 (+ 2613us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:06.269680 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:06.269683 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:06.269710 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:06.271166 (+ 1456us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.271662 (+ 496us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.271665 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.271666 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.271707 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.271905 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.271907 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.271908 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.272057 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:06.272167 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.272485 (+ 318us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.272644 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260131 03:02:06.276203 7369 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:02:06.279784 7369 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:02:06.294083 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.289488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:40273 (local address 127.0.0.1:34114)
0131 03:02:06.290527 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.290541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.290632 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.291126 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.291129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.291169 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:06.291458 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.291465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.292775 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.292778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.293570 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.293579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.293655 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.293709 (+ 54us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.293784 (+ 75us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.293872 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"spinlock_wait_cycles":2304,"thread_start_us":111,"threads_started":1}
I20260131 03:02:06.294087 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.289264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:43507 (local address 127.0.0.1:55772)
0131 03:02:06.290061 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.290100 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.290247 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.291114 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.291120 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.291141 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:06.291401 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.291408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.292731 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.292734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.293619 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.293627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.293748 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.293767 (+ 19us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.293817 (+ 50us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.293879 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":132,"threads_started":1}
I20260131 03:02:06.294750 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.289438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55772 (local address 127.6.22.66:43507)
0131 03:02:06.289962 (+ 524us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.289969 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.290199 (+ 230us) server_negotiation.cc:408] Connection header received
0131 03:02:06.290819 (+ 620us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.290823 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.290956 (+ 133us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.291113 (+ 157us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:06.291563 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.292561 (+ 998us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.293751 (+ 1190us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.294306 (+ 555us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.294455 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.294519 (+ 64us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.294609 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":106,"threads_started":1}
I20260131 03:02:06.295228 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.290408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:39139 (local address 127.0.0.1:52606)
0131 03:02:06.291376 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.291389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.291476 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.291898 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.291901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.291920 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:06.292169 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.292176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.293558 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.293562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.294844 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.294869 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.294984 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.295005 (+ 21us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.295055 (+ 50us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.295117 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":98,"threads_started":1}
I20260131 03:02:06.295922 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.290548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.6.22.67:39139)
0131 03:02:06.290905 (+ 357us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.290911 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.291481 (+ 570us) server_negotiation.cc:408] Connection header received
0131 03:02:06.291604 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.291612 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.291766 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.291892 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:06.292335 (+ 443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.293439 (+ 1104us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.295004 (+ 1565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.295546 (+ 542us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.295645 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.295704 (+ 59us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.295784 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260131 03:02:06.296361 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.289624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.6.22.65:40273)
0131 03:02:06.290541 (+ 917us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.290548 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.290599 (+ 51us) server_negotiation.cc:408] Connection header received
0131 03:02:06.290785 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.290791 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.290974 (+ 183us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.291118 (+ 144us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:06.291587 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.292619 (+ 1032us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.295456 (+ 2837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.295958 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.296066 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.296136 (+ 70us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.296231 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":60,"threads_started":1}
I20260131 03:02:06.297951 7651 tablet_service.cc:1505] Processing CreateTablet for tablet 9bab695f0bd740e29586eee7971ee864 (DEFAULT_TABLE table=test-table [id=df4dd9f4c74d4ddea8cc6af564955bb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:06.298312 7793 tablet_service.cc:1505] Processing CreateTablet for tablet 9bab695f0bd740e29586eee7971ee864 (DEFAULT_TABLE table=test-table [id=df4dd9f4c74d4ddea8cc6af564955bb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:06.299017 7651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bab695f0bd740e29586eee7971ee864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:06.298725 7507 tablet_service.cc:1505] Processing CreateTablet for tablet 9bab695f0bd740e29586eee7971ee864 (DEFAULT_TABLE table=test-table [id=df4dd9f4c74d4ddea8cc6af564955bb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:06.299201 7793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bab695f0bd740e29586eee7971ee864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:06.299557 7507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bab695f0bd740e29586eee7971ee864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:06.304445 7882 tablet_bootstrap.cc:492] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903: Bootstrap starting.
I20260131 03:02:06.305881 7883 tablet_bootstrap.cc:492] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368: Bootstrap starting.
I20260131 03:02:06.306749 7884 tablet_bootstrap.cc:492] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1: Bootstrap starting.
I20260131 03:02:06.307063 7882 tablet_bootstrap.cc:654] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:06.308028 7883 tablet_bootstrap.cc:654] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:06.308437 7884 tablet_bootstrap.cc:654] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:06.308439 7882 log.cc:826] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:06.309185 7883 log.cc:826] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:06.309266 7884 log.cc:826] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:06.310523 7882 tablet_bootstrap.cc:492] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903: No bootstrap required, opened a new log
I20260131 03:02:06.310719 7882 ts_tablet_manager.cc:1403] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260131 03:02:06.310854 7883 tablet_bootstrap.cc:492] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368: No bootstrap required, opened a new log
I20260131 03:02:06.310928 7884 tablet_bootstrap.cc:492] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1: No bootstrap required, opened a new log
I20260131 03:02:06.311084 7883 ts_tablet_manager.cc:1403] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260131 03:02:06.311105 7884 ts_tablet_manager.cc:1403] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260131 03:02:06.313710 7883 raft_consensus.cc:359] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.313942 7883 raft_consensus.cc:385] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:06.314003 7883 raft_consensus.cc:740] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362f6a72983a4499b326b3e277ba7368, State: Initialized, Role: FOLLOWER
I20260131 03:02:06.314492 7883 consensus_queue.cc:260] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.314455 7884 raft_consensus.cc:359] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.314453 7882 raft_consensus.cc:359] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.314776 7884 raft_consensus.cc:385] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:06.314776 7882 raft_consensus.cc:385] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:06.314846 7882 raft_consensus.cc:740] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c759e57501c44996a27cba6493bc2903, State: Initialized, Role: FOLLOWER
I20260131 03:02:06.314878 7884 raft_consensus.cc:740] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23a177155a34f22a836aa46035ad1f1, State: Initialized, Role: FOLLOWER
I20260131 03:02:06.315474 7883 ts_tablet_manager.cc:1434] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260131 03:02:06.315429 7882 consensus_queue.cc:260] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.315429 7884 consensus_queue.cc:260] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.315874 7717 heartbeater.cc:499] Master 127.6.22.126:38373 was elected leader, sending a full tablet report...
I20260131 03:02:06.316463 7884 ts_tablet_manager.cc:1434] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260131 03:02:06.316463 7882 ts_tablet_manager.cc:1434] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260131 03:02:06.316955 7575 heartbeater.cc:499] Master 127.6.22.126:38373 was elected leader, sending a full tablet report...
I20260131 03:02:06.318215 7859 heartbeater.cc:499] Master 127.6.22.126:38373 was elected leader, sending a full tablet report...
I20260131 03:02:06.394658 7889 raft_consensus.cc:493] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:02:06.394879 7889 raft_consensus.cc:515] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.396615 7889 leader_election.cc:290] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 362f6a72983a4499b326b3e277ba7368 (127.6.22.66:43507), c759e57501c44996a27cba6493bc2903 (127.6.22.67:39139)
I20260131 03:02:06.400596 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.397006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:39139 (local address 127.6.22.65:49785)
0131 03:02:06.397439 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.397451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.397596 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.397900 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.397906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.397925 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:06.398266 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.398273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.399359 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.399363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.400123 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.400132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.400228 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.400250 (+ 22us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.400290 (+ 40us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.400397 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"spinlock_wait_cycles":31488,"thread_start_us":156,"threads_started":1}
I20260131 03:02:06.401338 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.397136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:49785 (local address 127.6.22.67:39139)
0131 03:02:06.397393 (+ 257us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.397398 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.397564 (+ 166us) server_negotiation.cc:408] Connection header received
0131 03:02:06.397743 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.397746 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.397795 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.397858 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:06.398392 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.399203 (+ 811us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.400302 (+ 1099us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.400990 (+ 688us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.401040 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.401104 (+ 64us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.401181 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260131 03:02:06.401965 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.396902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:43507 (local address 127.6.22.65:60751)
0131 03:02:06.397377 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.397394 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.397573 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.397937 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.397940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.397998 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:06.398738 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.398746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.399745 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.399751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.400710 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.400728 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.401596 (+ 868us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.401619 (+ 23us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.401705 (+ 86us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.401792 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"spinlock_wait_cycles":768,"thread_start_us":216,"threads_started":1}
I20260131 03:02:06.401983 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.397162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:60751 (local address 127.6.22.66:43507)
0131 03:02:06.397352 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.397358 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.397509 (+ 151us) server_negotiation.cc:408] Connection header received
0131 03:02:06.397747 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.397750 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.397799 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.397920 (+ 121us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:06.398880 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.399564 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.400946 (+ 1382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.401471 (+ 525us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.401513 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.401768 (+ 255us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.401870 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260131 03:02:06.402194 7813 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bab695f0bd740e29586eee7971ee864" candidate_uuid: "b23a177155a34f22a836aa46035ad1f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c759e57501c44996a27cba6493bc2903" is_pre_election: true
I20260131 03:02:06.402513 7813 raft_consensus.cc:2468] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b23a177155a34f22a836aa46035ad1f1 in term 0.
I20260131 03:02:06.402760 7671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bab695f0bd740e29586eee7971ee864" candidate_uuid: "b23a177155a34f22a836aa46035ad1f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "362f6a72983a4499b326b3e277ba7368" is_pre_election: true
I20260131 03:02:06.403009 7671 raft_consensus.cc:2468] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b23a177155a34f22a836aa46035ad1f1 in term 0.
I20260131 03:02:06.403028 7460 leader_election.cc:304] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b23a177155a34f22a836aa46035ad1f1, c759e57501c44996a27cba6493bc2903; no voters:
I20260131 03:02:06.403354 7889 raft_consensus.cc:2804] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:02:06.403436 7889 raft_consensus.cc:493] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:02:06.403477 7889 raft_consensus.cc:3060] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:06.404457 7889 raft_consensus.cc:515] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.404906 7889 leader_election.cc:290] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [CANDIDATE]: Term 1 election: Requested vote from peers 362f6a72983a4499b326b3e277ba7368 (127.6.22.66:43507), c759e57501c44996a27cba6493bc2903 (127.6.22.67:39139)
I20260131 03:02:06.405239 7671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bab695f0bd740e29586eee7971ee864" candidate_uuid: "b23a177155a34f22a836aa46035ad1f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "362f6a72983a4499b326b3e277ba7368"
I20260131 03:02:06.405376 7671 raft_consensus.cc:3060] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:06.405362 7813 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bab695f0bd740e29586eee7971ee864" candidate_uuid: "b23a177155a34f22a836aa46035ad1f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c759e57501c44996a27cba6493bc2903"
I20260131 03:02:06.405505 7813 raft_consensus.cc:3060] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:06.406430 7671 raft_consensus.cc:2468] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b23a177155a34f22a836aa46035ad1f1 in term 1.
I20260131 03:02:06.406503 7813 raft_consensus.cc:2468] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b23a177155a34f22a836aa46035ad1f1 in term 1.
I20260131 03:02:06.406821 7460 leader_election.cc:304] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b23a177155a34f22a836aa46035ad1f1, c759e57501c44996a27cba6493bc2903; no voters:
I20260131 03:02:06.407054 7889 raft_consensus.cc:2804] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:06.407291 7889 raft_consensus.cc:697] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 1 LEADER]: Becoming Leader. State: Replica: b23a177155a34f22a836aa46035ad1f1, State: Running, Role: LEADER
I20260131 03:02:06.407624 7889 consensus_queue.cc:237] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.411000 7368 catalog_manager.cc:5654] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b23a177155a34f22a836aa46035ad1f1 (127.6.22.65). New cstate: current_term: 1 leader_uuid: "b23a177155a34f22a836aa46035ad1f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } health_report { overall_health: UNKNOWN } } }
I20260131 03:02:06.426708 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.423663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.6.22.65:40273)
0131 03:02:06.423908 (+ 245us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.423912 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.423925 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:02:06.424000 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.424003 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.424059 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.424142 (+ 83us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:06.424473 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.425026 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.425643 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.425778 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.425827 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:06.426292 (+ 465us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:06.426391 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.426467 (+ 76us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.426515 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260131 03:02:06.433107 7671 raft_consensus.cc:1275] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368 [term 1 FOLLOWER]: Refusing update from remote peer b23a177155a34f22a836aa46035ad1f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:06.433180 7813 raft_consensus.cc:1275] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903 [term 1 FOLLOWER]: Refusing update from remote peer b23a177155a34f22a836aa46035ad1f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:06.433938 7889 consensus_queue.cc:1048] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:06.434187 7893 consensus_queue.cc:1048] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:06.445894 7896 mvcc.cc:204] Tried to move back new op lower bound from 7249217644261154816 to 7249217644168298496. Current Snapshot: MvccSnapshot[applied={T|T < 7249217644261154816}]
I20260131 03:02:06.446951 7897 mvcc.cc:204] Tried to move back new op lower bound from 7249217644261154816 to 7249217644168298496. Current Snapshot: MvccSnapshot[applied={T|T < 7249217644261154816}]
I20260131 03:02:06.447544 7898 mvcc.cc:204] Tried to move back new op lower bound from 7249217644261154816 to 7249217644168298496. Current Snapshot: MvccSnapshot[applied={T|T < 7249217644261154816}]
I20260131 03:02:06.460356 7369 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:06.460572 7369 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:06.464038 7369 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8: Sending DeleteTablet for 3 replicas of tablet 9bab695f0bd740e29586eee7971ee864
I20260131 03:02:06.464993 7651 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bab695f0bd740e29586eee7971ee864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:06 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:55772
I20260131 03:02:06.465274 7507 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bab695f0bd740e29586eee7971ee864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:06 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:34114
I20260131 03:02:06.465276 7793 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bab695f0bd740e29586eee7971ee864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:06 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:52606
I20260131 03:02:06.465557 7905 tablet_replica.cc:333] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368: stopping tablet replica
I20260131 03:02:06.465870 7905 raft_consensus.cc:2243] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:06.466249 7905 raft_consensus.cc:2272] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:06.466419 7907 tablet_replica.cc:333] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1: stopping tablet replica
I20260131 03:02:06.466657 7907 raft_consensus.cc:2243] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:02:06.466717 7369 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260131 03:02:06.467093 7907 raft_consensus.cc:2272] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260131 03:02:06.467216 7369 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:02:06.467446 7906 tablet_replica.cc:333] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903: stopping tablet replica
I20260131 03:02:06.467685 7906 raft_consensus.cc:2243] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:06.467991 7906 raft_consensus.cc:2272] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:06.469435 7905 ts_tablet_manager.cc:1916] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:06.469493 7906 ts_tablet_manager.cc:1916] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:06.470444 7907 ts_tablet_manager.cc:1916] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:06.471941 7906 ts_tablet_manager.cc:1929] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:02:06.471941 7905 ts_tablet_manager.cc:1929] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:02:06.472038 7906 log.cc:1199] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/9bab695f0bd740e29586eee7971ee864
I20260131 03:02:06.472038 7905 log.cc:1199] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/9bab695f0bd740e29586eee7971ee864
I20260131 03:02:06.472364 7906 ts_tablet_manager.cc:1950] T 9bab695f0bd740e29586eee7971ee864 P c759e57501c44996a27cba6493bc2903: Deleting consensus metadata
I20260131 03:02:06.472376 7905 ts_tablet_manager.cc:1950] T 9bab695f0bd740e29586eee7971ee864 P 362f6a72983a4499b326b3e277ba7368: Deleting consensus metadata
I20260131 03:02:06.473642 7354 catalog_manager.cc:4985] TS c759e57501c44996a27cba6493bc2903 (127.6.22.67:39139): tablet 9bab695f0bd740e29586eee7971ee864 (table test-table [id=df4dd9f4c74d4ddea8cc6af564955bb5]) successfully deleted
I20260131 03:02:06.473871 7355 catalog_manager.cc:4985] TS 362f6a72983a4499b326b3e277ba7368 (127.6.22.66:43507): tablet 9bab695f0bd740e29586eee7971ee864 (table test-table [id=df4dd9f4c74d4ddea8cc6af564955bb5]) successfully deleted
I20260131 03:02:06.475654 7907 ts_tablet_manager.cc:1929] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:02:06.475766 7907 log.cc:1199] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/wal/wals/9bab695f0bd740e29586eee7971ee864
I20260131 03:02:06.476131 7907 ts_tablet_manager.cc:1950] T 9bab695f0bd740e29586eee7971ee864 P b23a177155a34f22a836aa46035ad1f1: Deleting consensus metadata
I20260131 03:02:06.476423 7651 tablet_service.cc:1505] Processing CreateTablet for tablet 5f592aeaf2fa4e388d31e8d8516ae35b (DEFAULT_TABLE table=test-table [id=1621bade9e1e48458ff698679ba77e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:06.476456 7793 tablet_service.cc:1505] Processing CreateTablet for tablet 5f592aeaf2fa4e388d31e8d8516ae35b (DEFAULT_TABLE table=test-table [id=1621bade9e1e48458ff698679ba77e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:06.476696 7651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f592aeaf2fa4e388d31e8d8516ae35b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:06.476742 7793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f592aeaf2fa4e388d31e8d8516ae35b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:06.478850 7883 tablet_bootstrap.cc:492] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368: Bootstrap starting.
I20260131 03:02:06.475718 7507 tablet_service.cc:1505] Processing CreateTablet for tablet 5f592aeaf2fa4e388d31e8d8516ae35b (DEFAULT_TABLE table=test-table [id=1621bade9e1e48458ff698679ba77e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:06.479528 7507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f592aeaf2fa4e388d31e8d8516ae35b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:06.479619 7882 tablet_bootstrap.cc:492] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903: Bootstrap starting.
I20260131 03:02:06.479897 7883 tablet_bootstrap.cc:654] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:06.480834 7882 tablet_bootstrap.cc:654] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:06.480875 7357 catalog_manager.cc:4985] TS b23a177155a34f22a836aa46035ad1f1 (127.6.22.65:40273): tablet 9bab695f0bd740e29586eee7971ee864 (table test-table [id=df4dd9f4c74d4ddea8cc6af564955bb5]) successfully deleted
I20260131 03:02:06.481854 7883 tablet_bootstrap.cc:492] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368: No bootstrap required, opened a new log
I20260131 03:02:06.481945 7883 ts_tablet_manager.cc:1403] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260131 03:02:06.482441 7883 raft_consensus.cc:359] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.482548 7883 raft_consensus.cc:385] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:06.482579 7883 raft_consensus.cc:740] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362f6a72983a4499b326b3e277ba7368, State: Initialized, Role: FOLLOWER
I20260131 03:02:06.482710 7883 consensus_queue.cc:260] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.482901 7884 tablet_bootstrap.cc:492] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1: Bootstrap starting.
I20260131 03:02:06.482946 7883 ts_tablet_manager.cc:1434] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:06.484074 7884 tablet_bootstrap.cc:654] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:06.484759 7882 tablet_bootstrap.cc:492] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903: No bootstrap required, opened a new log
I20260131 03:02:06.484887 7882 ts_tablet_manager.cc:1403] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260131 03:02:06.485359 7882 raft_consensus.cc:359] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.485491 7882 raft_consensus.cc:385] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:06.485540 7882 raft_consensus.cc:740] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c759e57501c44996a27cba6493bc2903, State: Initialized, Role: FOLLOWER
I20260131 03:02:06.485743 7882 consensus_queue.cc:260] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.486047 7882 ts_tablet_manager.cc:1434] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260131 03:02:06.487131 7884 tablet_bootstrap.cc:492] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1: No bootstrap required, opened a new log
I20260131 03:02:06.487244 7884 ts_tablet_manager.cc:1403] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260131 03:02:06.487761 7884 raft_consensus.cc:359] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.487884 7884 raft_consensus.cc:385] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:06.487923 7884 raft_consensus.cc:740] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23a177155a34f22a836aa46035ad1f1, State: Initialized, Role: FOLLOWER
I20260131 03:02:06.488078 7884 consensus_queue.cc:260] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.488384 7884 ts_tablet_manager.cc:1434] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260131 03:02:06.490126 7718 tablet.cc:2378] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:02:06.558266 7888 raft_consensus.cc:493] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:02:06.558441 7888 raft_consensus.cc:515] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.559777 7888 leader_election.cc:290] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b23a177155a34f22a836aa46035ad1f1 (127.6.22.65:40273), c759e57501c44996a27cba6493bc2903 (127.6.22.67:39139)
I20260131 03:02:06.563728 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.560029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:39139 (local address 127.6.22.66:43335)
0131 03:02:06.560627 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.560641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.560825 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.561211 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.561213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.561260 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:06.561483 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.561488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.562568 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.562571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.563280 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.563296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.563421 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.563444 (+ 23us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.563499 (+ 55us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.563562 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":278,"spinlock_wait_cycles":52992,"thread_start_us":70,"threads_started":1}
I20260131 03:02:06.564157 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.560140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:43335 (local address 127.6.22.67:39139)
0131 03:02:06.560277 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.560281 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.560735 (+ 454us) server_negotiation.cc:408] Connection header received
0131 03:02:06.560948 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.560952 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.561011 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.561064 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:06.561601 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.562403 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.563413 (+ 1010us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.563870 (+ 457us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.563922 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.563973 (+ 51us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.564043 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260131 03:02:06.564524 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.560355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:44121 (local address 127.6.22.65:40273)
0131 03:02:06.560494 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.560499 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.560722 (+ 223us) server_negotiation.cc:408] Connection header received
0131 03:02:06.560943 (+ 221us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.560947 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.560994 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.561041 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:06.561670 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.562403 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.563634 (+ 1231us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.564102 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.564139 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.564344 (+ 205us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.564408 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260131 03:02:06.564608 7813 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f592aeaf2fa4e388d31e8d8516ae35b" candidate_uuid: "362f6a72983a4499b326b3e277ba7368" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c759e57501c44996a27cba6493bc2903" is_pre_election: true
I20260131 03:02:06.564793 7813 raft_consensus.cc:2468] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362f6a72983a4499b326b3e277ba7368 in term 0.
I20260131 03:02:06.564872 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.560029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:40273 (local address 127.6.22.66:44121)
0131 03:02:06.560605 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.560619 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.560816 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.561192 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.561195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.561214 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:06.561514 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.561521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.562528 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.562532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.563500 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.563510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.564200 (+ 690us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.564215 (+ 15us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.564678 (+ 463us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.564742 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"spinlock_wait_cycles":4224,"thread_start_us":89,"threads_started":1}
I20260131 03:02:06.565227 7602 leader_election.cc:304] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362f6a72983a4499b326b3e277ba7368, c759e57501c44996a27cba6493bc2903; no voters:
I20260131 03:02:06.565469 7888 raft_consensus.cc:2804] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:02:06.565485 7525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f592aeaf2fa4e388d31e8d8516ae35b" candidate_uuid: "362f6a72983a4499b326b3e277ba7368" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23a177155a34f22a836aa46035ad1f1" is_pre_election: true
I20260131 03:02:06.565544 7888 raft_consensus.cc:493] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:02:06.565586 7888 raft_consensus.cc:3060] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:06.565682 7525 raft_consensus.cc:2468] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362f6a72983a4499b326b3e277ba7368 in term 0.
I20260131 03:02:06.566663 7888 raft_consensus.cc:515] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.567080 7888 leader_election.cc:290] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [CANDIDATE]: Term 1 election: Requested vote from peers b23a177155a34f22a836aa46035ad1f1 (127.6.22.65:40273), c759e57501c44996a27cba6493bc2903 (127.6.22.67:39139)
I20260131 03:02:06.567381 7525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f592aeaf2fa4e388d31e8d8516ae35b" candidate_uuid: "362f6a72983a4499b326b3e277ba7368" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23a177155a34f22a836aa46035ad1f1"
I20260131 03:02:06.567413 7813 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f592aeaf2fa4e388d31e8d8516ae35b" candidate_uuid: "362f6a72983a4499b326b3e277ba7368" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c759e57501c44996a27cba6493bc2903"
I20260131 03:02:06.567490 7525 raft_consensus.cc:3060] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:06.567533 7813 raft_consensus.cc:3060] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:06.568396 7813 raft_consensus.cc:2468] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362f6a72983a4499b326b3e277ba7368 in term 1.
I20260131 03:02:06.568693 7525 raft_consensus.cc:2468] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362f6a72983a4499b326b3e277ba7368 in term 1.
I20260131 03:02:06.568742 7602 leader_election.cc:304] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362f6a72983a4499b326b3e277ba7368, c759e57501c44996a27cba6493bc2903; no voters:
I20260131 03:02:06.568928 7888 raft_consensus.cc:2804] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:06.569134 7888 raft_consensus.cc:697] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 1 LEADER]: Becoming Leader. State: Replica: 362f6a72983a4499b326b3e277ba7368, State: Running, Role: LEADER
I20260131 03:02:06.569387 7888 consensus_queue.cc:237] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.571704 7369 catalog_manager.cc:5654] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 reported cstate change: term changed from 0 to 1, leader changed from <none> to 362f6a72983a4499b326b3e277ba7368 (127.6.22.66). New cstate: current_term: 1 leader_uuid: "362f6a72983a4499b326b3e277ba7368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } health_report { overall_health: UNKNOWN } } }
Jan 31 03:02:06 dist-test-slave-j368 krb5kdc[7322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:06.608526 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.599187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38373 (local address 127.0.0.1:43604)
0131 03:02:06.599537 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.599556 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.599709 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.600152 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.600156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.600782 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:06.601005 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.601013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.602115 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.602118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.602492 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.602499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.602592 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.603196 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:06.603210 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:06.605023 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:06.606902 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:06.606914 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:06.606923 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:06.607203 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:06.607481 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:06.607483 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:06.607485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:06.607532 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:06.607918 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:06.607923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:06.608020 (+ 97us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.608115 (+ 95us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.608280 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":137,"threads_started":1}
I20260131 03:02:06.608639 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.599376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43604 (local address 127.6.22.126:38373)
0131 03:02:06.599614 (+ 238us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.599620 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.599649 (+ 29us) server_negotiation.cc:408] Connection header received
0131 03:02:06.599836 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.599840 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.599908 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.600046 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:06.601157 (+ 1111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.601979 (+ 822us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.602609 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.602786 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.605158 (+ 2372us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:06.605202 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:06.605205 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:06.605233 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:06.606761 (+ 1528us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.607321 (+ 560us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.607325 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.607327 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.607385 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.607618 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.607624 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.607626 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.607817 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:06.607916 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.608313 (+ 397us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.608442 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260131 03:02:06.613121 7369 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:43604:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260131 03:02:06.613969 7369 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:02:06.620321 7507 tablet_service.cc:1505] Processing CreateTablet for tablet 67f30fe91454460ea93a85dceb4919dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbe30bd4bb894e2191780ceafe7748ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:02:06.620785 7507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67f30fe91454460ea93a85dceb4919dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:06.620720 7651 tablet_service.cc:1505] Processing CreateTablet for tablet 67f30fe91454460ea93a85dceb4919dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbe30bd4bb894e2191780ceafe7748ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:02:06.621280 7651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67f30fe91454460ea93a85dceb4919dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:06.623100 7884 tablet_bootstrap.cc:492] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: Bootstrap starting.
I20260131 03:02:06.623459 7793 tablet_service.cc:1505] Processing CreateTablet for tablet 67f30fe91454460ea93a85dceb4919dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbe30bd4bb894e2191780ceafe7748ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:02:06.623828 7793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67f30fe91454460ea93a85dceb4919dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:06.624100 7883 tablet_bootstrap.cc:492] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368: Bootstrap starting.
I20260131 03:02:06.624217 7884 tablet_bootstrap.cc:654] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:06.625149 7883 tablet_bootstrap.cc:654] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:06.626492 7882 tablet_bootstrap.cc:492] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903: Bootstrap starting.
I20260131 03:02:06.626514 7883 tablet_bootstrap.cc:492] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368: No bootstrap required, opened a new log
I20260131 03:02:06.626595 7883 ts_tablet_manager.cc:1403] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260131 03:02:06.627092 7883 raft_consensus.cc:359] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.627223 7883 raft_consensus.cc:385] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:06.627259 7883 raft_consensus.cc:740] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362f6a72983a4499b326b3e277ba7368, State: Initialized, Role: FOLLOWER
I20260131 03:02:06.627400 7883 consensus_queue.cc:260] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.627583 7882 tablet_bootstrap.cc:654] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:06.627668 7888 tablet_replica.cc:442] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } } }
I20260131 03:02:06.627801 7888 tablet_replica.cc:445] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:06.627673 7883 ts_tablet_manager.cc:1434] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260131 03:02:06.628851 7884 tablet_bootstrap.cc:492] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: No bootstrap required, opened a new log
I20260131 03:02:06.628940 7884 ts_tablet_manager.cc:1403] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260131 03:02:06.629470 7884 raft_consensus.cc:359] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.629573 7884 raft_consensus.cc:385] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:06.629601 7884 raft_consensus.cc:740] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23a177155a34f22a836aa46035ad1f1, State: Initialized, Role: FOLLOWER
I20260131 03:02:06.629720 7884 consensus_queue.cc:260] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.629971 7884 ts_tablet_manager.cc:1434] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260131 03:02:06.630201 7889 tablet_replica.cc:442] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } } }
I20260131 03:02:06.630295 7889 tablet_replica.cc:445] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:06.630884 7882 tablet_bootstrap.cc:492] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903: No bootstrap required, opened a new log
I20260131 03:02:06.630972 7882 ts_tablet_manager.cc:1403] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260131 03:02:06.631582 7882 raft_consensus.cc:359] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.631692 7882 raft_consensus.cc:385] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:06.631732 7882 raft_consensus.cc:740] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c759e57501c44996a27cba6493bc2903, State: Initialized, Role: FOLLOWER
I20260131 03:02:06.631963 7882 consensus_queue.cc:260] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.632248 7890 tablet_replica.cc:442] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } } }
I20260131 03:02:06.632400 7890 tablet_replica.cc:445] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:06.633097 7882 ts_tablet_manager.cc:1434] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260131 03:02:06.730979 7576 tablet.cc:2378] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:02:06.735306 7860 tablet.cc:2378] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:02:06.750470 7889 raft_consensus.cc:493] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:02:06.750631 7889 raft_consensus.cc:515] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.751147 7889 leader_election.cc:290] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 362f6a72983a4499b326b3e277ba7368 (127.6.22.66:43507), c759e57501c44996a27cba6493bc2903 (127.6.22.67:39139)
I20260131 03:02:06.751643 7813 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67f30fe91454460ea93a85dceb4919dd" candidate_uuid: "b23a177155a34f22a836aa46035ad1f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c759e57501c44996a27cba6493bc2903" is_pre_election: true
I20260131 03:02:06.751642 7671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67f30fe91454460ea93a85dceb4919dd" candidate_uuid: "b23a177155a34f22a836aa46035ad1f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "362f6a72983a4499b326b3e277ba7368" is_pre_election: true
I20260131 03:02:06.751809 7813 raft_consensus.cc:2468] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b23a177155a34f22a836aa46035ad1f1 in term 0.
I20260131 03:02:06.751816 7671 raft_consensus.cc:2468] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b23a177155a34f22a836aa46035ad1f1 in term 0.
I20260131 03:02:06.752141 7460 leader_election.cc:304] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b23a177155a34f22a836aa46035ad1f1, c759e57501c44996a27cba6493bc2903; no voters:
I20260131 03:02:06.752359 7889 raft_consensus.cc:2804] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:02:06.752430 7889 raft_consensus.cc:493] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:02:06.752475 7889 raft_consensus.cc:3060] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:06.753535 7889 raft_consensus.cc:515] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.753927 7889 leader_election.cc:290] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [CANDIDATE]: Term 1 election: Requested vote from peers 362f6a72983a4499b326b3e277ba7368 (127.6.22.66:43507), c759e57501c44996a27cba6493bc2903 (127.6.22.67:39139)
I20260131 03:02:06.754228 7671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67f30fe91454460ea93a85dceb4919dd" candidate_uuid: "b23a177155a34f22a836aa46035ad1f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "362f6a72983a4499b326b3e277ba7368"
I20260131 03:02:06.754362 7671 raft_consensus.cc:3060] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:06.754364 7813 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67f30fe91454460ea93a85dceb4919dd" candidate_uuid: "b23a177155a34f22a836aa46035ad1f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c759e57501c44996a27cba6493bc2903"
I20260131 03:02:06.754477 7813 raft_consensus.cc:3060] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:06.755314 7671 raft_consensus.cc:2468] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b23a177155a34f22a836aa46035ad1f1 in term 1.
I20260131 03:02:06.755506 7813 raft_consensus.cc:2468] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b23a177155a34f22a836aa46035ad1f1 in term 1.
I20260131 03:02:06.755645 7461 leader_election.cc:304] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362f6a72983a4499b326b3e277ba7368, b23a177155a34f22a836aa46035ad1f1; no voters:
I20260131 03:02:06.755851 7889 raft_consensus.cc:2804] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:06.755955 7889 raft_consensus.cc:697] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [term 1 LEADER]: Becoming Leader. State: Replica: b23a177155a34f22a836aa46035ad1f1, State: Running, Role: LEADER
I20260131 03:02:06.756099 7889 consensus_queue.cc:237] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } }
I20260131 03:02:06.756808 7893 tablet_replica.cc:442] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: TxnStatusTablet state changed. Reason: New leader b23a177155a34f22a836aa46035ad1f1. Latest consensus state: current_term: 1 leader_uuid: "b23a177155a34f22a836aa46035ad1f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } } }
I20260131 03:02:06.756907 7893 tablet_replica.cc:445] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:02:06.757162 7925 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260131 03:02:06.757298 7925 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260131 03:02:06.757905 7368 catalog_manager.cc:5654] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b23a177155a34f22a836aa46035ad1f1 (127.6.22.65). New cstate: current_term: 1 leader_uuid: "b23a177155a34f22a836aa46035ad1f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } health_report { overall_health: UNKNOWN } } }
I20260131 03:02:06.790212 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.787446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:40273 (local address 127.0.0.1:34144)
0131 03:02:06.787610 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.787622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.787731 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.788098 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.788101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.788108 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:02:06.788292 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.788298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.789004 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.789008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.789404 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.789411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.789528 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.789573 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:02:06.789915 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:02:06.789934 (+ 19us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.790018 (+ 84us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.790093 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260131 03:02:06.790212 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.787529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.6.22.65:40273)
0131 03:02:06.787889 (+ 360us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.787892 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.787902 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:02:06.787942 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.787944 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.787987 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.788073 (+ 86us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:06.788397 (+ 324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.788901 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.789514 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.789725 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.789765 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:06.789825 (+ 60us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:06.789907 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.790030 (+ 123us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.790081 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290}
I20260131 03:02:06.793004 7813 raft_consensus.cc:1275] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903 [term 1 FOLLOWER]: Refusing update from remote peer b23a177155a34f22a836aa46035ad1f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:06.793041 7671 raft_consensus.cc:1275] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368 [term 1 FOLLOWER]: Refusing update from remote peer b23a177155a34f22a836aa46035ad1f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:06.793458 7893 consensus_queue.cc:1048] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:06.793603 7889 consensus_queue.cc:1048] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:06.794867 7890 tablet_replica.cc:442] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903: TxnStatusTablet state changed. Reason: New leader b23a177155a34f22a836aa46035ad1f1. Latest consensus state: current_term: 1 leader_uuid: "b23a177155a34f22a836aa46035ad1f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } } }
I20260131 03:02:06.794987 7890 tablet_replica.cc:445] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:06.795089 7888 tablet_replica.cc:442] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368: TxnStatusTablet state changed. Reason: New leader b23a177155a34f22a836aa46035ad1f1. Latest consensus state: current_term: 1 leader_uuid: "b23a177155a34f22a836aa46035ad1f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } } }
I20260131 03:02:06.795195 7888 tablet_replica.cc:445] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:06.797147 7889 tablet_replica.cc:442] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b23a177155a34f22a836aa46035ad1f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } } }
I20260131 03:02:06.797236 7889 tablet_replica.cc:445] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:02:06.797633 7888 tablet_replica.cc:442] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b23a177155a34f22a836aa46035ad1f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } } }
I20260131 03:02:06.797708 7888 tablet_replica.cc:445] T 67f30fe91454460ea93a85dceb4919dd P 362f6a72983a4499b326b3e277ba7368: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:06.798736 7893 tablet_replica.cc:442] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b23a177155a34f22a836aa46035ad1f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } } }
I20260131 03:02:06.798844 7893 tablet_replica.cc:445] T 67f30fe91454460ea93a85dceb4919dd P b23a177155a34f22a836aa46035ad1f1: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:02:06.799608 7890 tablet_replica.cc:442] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b23a177155a34f22a836aa46035ad1f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 } } peers { permanent_uuid: "362f6a72983a4499b326b3e277ba7368" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 43507 } } peers { permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 } } }
I20260131 03:02:06.799702 7890 tablet_replica.cc:445] T 67f30fe91454460ea93a85dceb4919dd P c759e57501c44996a27cba6493bc2903: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:06.805003 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.801755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.6.22.66:43507)
0131 03:02:06.801854 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.801858 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.801924 (+ 66us) server_negotiation.cc:408] Connection header received
0131 03:02:06.802089 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.802094 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.802157 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.802242 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:06.802608 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.803213 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.803981 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.804135 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.804181 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:06.804596 (+ 415us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:06.804774 (+ 178us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.804839 (+ 65us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.804888 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:02:06.814553 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.811952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:59789 (local address 127.6.22.65:40273)
0131 03:02:06.812106 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.812118 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.812314 (+ 196us) server_negotiation.cc:408] Connection header received
0131 03:02:06.812452 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.812455 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.812501 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.812589 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:06.812961 (+ 372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.813380 (+ 419us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.813956 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.814128 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.814164 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:06.814218 (+ 54us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:06.814298 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.814394 (+ 96us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.814434 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260131 03:02:06.814581 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.811873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:40273 (local address 127.6.22.66:59789)
0131 03:02:06.812231 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.812241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.812363 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.812612 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.812615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.812621 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:02:06.812862 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.812867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.813488 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.813491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.813855 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.813861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.813968 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.814006 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:02:06.814298 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:02:06.814304 (+ 6us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.814384 (+ 80us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.814442 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":120,"threads_started":1}
I20260131 03:02:06.821138 7813 raft_consensus.cc:1275] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903 [term 1 FOLLOWER]: Refusing update from remote peer 362f6a72983a4499b326b3e277ba7368: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:06.821206 7525 raft_consensus.cc:1275] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1 [term 1 FOLLOWER]: Refusing update from remote peer 362f6a72983a4499b326b3e277ba7368: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:06.821615 7888 consensus_queue.cc:1048] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b23a177155a34f22a836aa46035ad1f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:06.821767 7913 consensus_queue.cc:1048] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c759e57501c44996a27cba6493bc2903" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 39139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:06.843348 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.838799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:44517 (local address 127.6.22.66:43507)
0131 03:02:06.838905 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.838908 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.840985 (+ 2077us) server_negotiation.cc:408] Connection header received
0131 03:02:06.841142 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.841146 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.841193 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.841266 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:06.841589 (+ 323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.842006 (+ 417us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.842577 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.842752 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.842786 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:06.842867 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:06.842934 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.843185 (+ 251us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.843227 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:02:06.843367 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.838740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:43507 (local address 127.6.22.65:44517)
0131 03:02:06.840905 (+ 2165us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:06.840916 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:06.841051 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:06.841276 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:06.841279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:06.841294 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:02:06.841493 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.841498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.842112 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.842116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:06.842474 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:06.842481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.842588 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.842639 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:02:06.843081 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:02:06.843088 (+ 7us) client_negotiation.cc:769] Sending connection context
0131 03:02:06.843163 (+ 75us) client_negotiation.cc:240] Negotiation successful
0131 03:02:06.843227 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2035,"thread_start_us":99,"threads_started":1}
I20260131 03:02:06.867139 7369 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:06.867290 7369 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:06.868991 7369 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 77cb500bb87a4c94b55816101052b3e8: Sending DeleteTablet for 3 replicas of tablet 5f592aeaf2fa4e388d31e8d8516ae35b
I20260131 03:02:06.869632 7793 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f592aeaf2fa4e388d31e8d8516ae35b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:06 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:52606
I20260131 03:02:06.869845 7906 tablet_replica.cc:333] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903: stopping tablet replica
I20260131 03:02:06.869969 7906 raft_consensus.cc:2243] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:06.870075 7906 raft_consensus.cc:2272] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:06.870599 7507 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f592aeaf2fa4e388d31e8d8516ae35b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:06 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:34114
I20260131 03:02:06.870922 7906 ts_tablet_manager.cc:1916] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:06.870929 7651 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f592aeaf2fa4e388d31e8d8516ae35b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:06 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:55772
I20260131 03:02:06.871083 7905 tablet_replica.cc:333] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368: stopping tablet replica
I20260131 03:02:06.871168 7905 raft_consensus.cc:2243] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:02:06.871551 7905 raft_consensus.cc:2272] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:06.872371 7907 tablet_replica.cc:333] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1: stopping tablet replica
I20260131 03:02:06.872416 7905 ts_tablet_manager.cc:1916] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:06.872480 7907 raft_consensus.cc:2243] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:06.872574 7907 raft_consensus.cc:2272] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:06.873317 7907 ts_tablet_manager.cc:1916] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:06.874759 7905 ts_tablet_manager.cc:1929] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:02:06.874815 7905 log.cc:1199] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/5f592aeaf2fa4e388d31e8d8516ae35b
I20260131 03:02:06.875097 7905 ts_tablet_manager.cc:1950] T 5f592aeaf2fa4e388d31e8d8516ae35b P 362f6a72983a4499b326b3e277ba7368: Deleting consensus metadata
I20260131 03:02:06.875669 7907 ts_tablet_manager.cc:1929] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:02:06.875741 7907 log.cc:1199] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-0/wal/wals/5f592aeaf2fa4e388d31e8d8516ae35b
I20260131 03:02:06.875732 7355 catalog_manager.cc:4985] TS 362f6a72983a4499b326b3e277ba7368 (127.6.22.66:43507): tablet 5f592aeaf2fa4e388d31e8d8516ae35b (table test-table [id=1621bade9e1e48458ff698679ba77e7d]) successfully deleted
I20260131 03:02:06.875988 7907 ts_tablet_manager.cc:1950] T 5f592aeaf2fa4e388d31e8d8516ae35b P b23a177155a34f22a836aa46035ad1f1: Deleting consensus metadata
I20260131 03:02:06.876602 7357 catalog_manager.cc:4985] TS b23a177155a34f22a836aa46035ad1f1 (127.6.22.65:40273): tablet 5f592aeaf2fa4e388d31e8d8516ae35b (table test-table [id=1621bade9e1e48458ff698679ba77e7d]) successfully deleted
I20260131 03:02:06.878475 7906 ts_tablet_manager.cc:1929] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:02:06.878551 7906 log.cc:1199] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/5f592aeaf2fa4e388d31e8d8516ae35b
I20260131 03:02:06.878793 7906 ts_tablet_manager.cc:1950] T 5f592aeaf2fa4e388d31e8d8516ae35b P c759e57501c44996a27cba6493bc2903: Deleting consensus metadata
I20260131 03:02:06.879488 7354 catalog_manager.cc:4985] TS c759e57501c44996a27cba6493bc2903 (127.6.22.67:39139): tablet 5f592aeaf2fa4e388d31e8d8516ae35b (table test-table [id=1621bade9e1e48458ff698679ba77e7d]) successfully deleted
Jan 31 03:02:06 dist-test-slave-j368 krb5kdc[7322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.22.65@KRBTEST.COM
I20260131 03:02:06.884891 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.871107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.6.22.65:40273)
0131 03:02:06.871209 (+ 102us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.871213 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.871896 (+ 683us) server_negotiation.cc:408] Connection header received
0131 03:02:06.872075 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.872080 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.872125 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.872176 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:06.873198 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.873769 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.874506 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.874683 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.881987 (+ 7304us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:06.882011 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:06.882020 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:06.882040 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:06.883442 (+ 1402us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.883808 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.883812 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.883814 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.883858 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.884108 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.884111 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.884113 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.884270 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:06.884452 (+ 182us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.884632 (+ 180us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.884764 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260131 03:02:06.885443 7556 server_base.cc:1142] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34152
I20260131 03:02:06.892977 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:06.886777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.6.22.126:38373)
0131 03:02:06.886882 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:06.886886 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:06.887187 (+ 301us) server_negotiation.cc:408] Connection header received
0131 03:02:06.887349 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:06.887352 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:06.887396 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:06.887485 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:06.888159 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.888730 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:06.889232 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:06.889676 (+ 444us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:06.890333 (+ 657us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:06.890349 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:06.890352 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:06.890376 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:06.891701 (+ 1325us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.892019 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.892022 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.892024 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.892074 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:06.892297 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:06.892301 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:06.892302 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:06.892428 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:06.892549 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:06.892751 (+ 202us) server_negotiation.cc:300] Negotiation successful
0131 03:02:06.892844 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260131 03:02:06.893402 7369 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43608
I20260131 03:02:06.894474 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 7444
I20260131 03:02:06.902453 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 7586
I20260131 03:02:06.909689 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 7728
I20260131 03:02:06.914867 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 7338
2026-01-31T03:02:06Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4143 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:02:06 dist-test-slave-j368 krb5kdc[7946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:02:06 dist-test-slave-j368 krb5kdc[7946](info): set up 2 sockets
Jan 31 03:02:06 dist-test-slave-j368 krb5kdc[7946](info): commencing operation
krb5kdc: starting...
W20260131 03:02:08.729990 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.793s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:02:08 dist-test-slave-j368 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:02:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:08.844841 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:45529
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:45529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:08.939496 7962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:08.939692 7962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:08.939740 7962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:08.942939 7962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:08.942988 7962 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260131 03:02:08.943006 7962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:08.943027 7962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:08.943089 7962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:08.943150 7962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:08.947074 7962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:45529
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:45529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:08.947906 7962 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:08.948664 7962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:08.953943 7967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:08.953948 7970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:08.954037 7968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:08.954128 7962 server_base.cc:1060] running on GCE node
I20260131 03:02:08.954658 7962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:08.955453 7962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:08.956625 7962 hybrid_clock.cc:648] HybridClock initialized: now 1769828528956605 us; error 41 us; skew 500 ppm
Jan 31 03:02:08 dist-test-slave-j368 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828528, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:08.958961 7962 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:02:08.960034 7962 webserver.cc:492] Webserver started at http://127.6.22.126:40859/ using document root <none> and password file <none>
I20260131 03:02:08.960530 7962 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:08.960602 7962 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:08.960817 7962 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:08.962302 7962 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "4f68701b7f7d40dfbcd97e4239095480"
format_stamp: "Formatted at 2026-01-31 03:02:08 on dist-test-slave-j368"
server_key: "2b1b5e6e006c88d3034ae00287f21e2b"
server_key_iv: "e87c4c8a33f57a5b670bd888bc34f274"
server_key_version: "encryptionkey@0"
I20260131 03:02:08.962760 7962 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "4f68701b7f7d40dfbcd97e4239095480"
format_stamp: "Formatted at 2026-01-31 03:02:08 on dist-test-slave-j368"
server_key: "2b1b5e6e006c88d3034ae00287f21e2b"
server_key_iv: "e87c4c8a33f57a5b670bd888bc34f274"
server_key_version: "encryptionkey@0"
I20260131 03:02:08.965999 7962 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260131 03:02:08.968297 7977 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:08.969651 7962 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260131 03:02:08.969790 7962 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "4f68701b7f7d40dfbcd97e4239095480"
format_stamp: "Formatted at 2026-01-31 03:02:08 on dist-test-slave-j368"
server_key: "2b1b5e6e006c88d3034ae00287f21e2b"
server_key_iv: "e87c4c8a33f57a5b670bd888bc34f274"
server_key_version: "encryptionkey@0"
I20260131 03:02:08.969892 7962 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:09.000447 7962 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:09.004025 7962 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:09.004349 7962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:09.016152 7962 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:45529
I20260131 03:02:09.016145 8039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:45529 every 8 connection(s)
I20260131 03:02:09.018008 7962 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:09.020565 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 7962
I20260131 03:02:09.020748 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:09.021010 6233 external_mini_cluster.cc:1468] Setting key 013174442a46a2f92960ca28add83401
I20260131 03:02:09.022562 8040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jan 31 03:02:09 dist-test-slave-j368 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:09.031968 8040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480: Bootstrap starting.
I20260131 03:02:09.035032 8040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:09.038070 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.022294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.6.22.126:45529)
0131 03:02:09.022705 (+ 411us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.022717 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.022756 (+ 39us) server_negotiation.cc:408] Connection header received
0131 03:02:09.023574 (+ 818us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.023599 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.024169 (+ 570us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.024576 (+ 407us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:09.025824 (+ 1248us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.027197 (+ 1373us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.028172 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.028467 (+ 295us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.031730 (+ 3263us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:09.031792 (+ 62us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:09.031809 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:09.031863 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:09.034345 (+ 2482us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.034862 (+ 517us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.034868 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.034874 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.034998 (+ 124us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.035293 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.035297 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.035299 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.035841 (+ 542us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:09.036095 (+ 254us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.036850 (+ 755us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.037166 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260131 03:02:09.038093 8040 log.cc:826] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:09.040035 8040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480: No bootstrap required, opened a new log
I20260131 03:02:09.043669 8040 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f68701b7f7d40dfbcd97e4239095480" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 45529 } }
I20260131 03:02:09.043893 8040 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:09.043946 8040 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f68701b7f7d40dfbcd97e4239095480, State: Initialized, Role: FOLLOWER
I20260131 03:02:09.044409 8040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f68701b7f7d40dfbcd97e4239095480" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 45529 } }
I20260131 03:02:09.044598 8040 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:09.044680 8040 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:09.044767 8040 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:09.045595 8040 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f68701b7f7d40dfbcd97e4239095480" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 45529 } }
I20260131 03:02:09.045943 8040 leader_election.cc:304] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f68701b7f7d40dfbcd97e4239095480; no voters:
I20260131 03:02:09.046240 8040 leader_election.cc:290] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:09.046375 8045 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:09.046605 8045 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [term 1 LEADER]: Becoming Leader. State: Replica: 4f68701b7f7d40dfbcd97e4239095480, State: Running, Role: LEADER
I20260131 03:02:09.046895 8045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f68701b7f7d40dfbcd97e4239095480" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 45529 } }
I20260131 03:02:09.047359 8040 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:09.048573 8047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f68701b7f7d40dfbcd97e4239095480. Latest consensus state: current_term: 1 leader_uuid: "4f68701b7f7d40dfbcd97e4239095480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f68701b7f7d40dfbcd97e4239095480" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 45529 } } }
I20260131 03:02:09.048573 8046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f68701b7f7d40dfbcd97e4239095480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f68701b7f7d40dfbcd97e4239095480" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 45529 } } }
I20260131 03:02:09.048833 8046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:09.048863 8047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:09.050052 8054 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:09.054347 8054 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:09.060062 8054 catalog_manager.cc:1357] Generated new cluster ID: 76e05d6ffeee4596a8d79001063c8150
I20260131 03:02:09.060166 8054 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:09.077749 8054 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:09.078826 8054 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:09.089432 8054 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480: Generated new TSK 0
I20260131 03:02:09.090812 8054 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:09.152087 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:45529
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:09.262920 8068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:09.263140 8068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:09.263216 8068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:09.266363 8068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:09.266424 8068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:09.266477 8068 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260131 03:02:09.266492 8068 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:02:09.266532 8068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:02:09.270676 8068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.22.126:45529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:09.271718 8068 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:09.272521 8068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:09.279302 8073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:09.279381 8068 server_base.cc:1060] running on GCE node
W20260131 03:02:09.279302 8074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:09.279305 8076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:09.279852 8068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:09.280427 8068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:09.281603 8068 hybrid_clock.cc:648] HybridClock initialized: now 1769828529281575 us; error 49 us; skew 500 ppm
Jan 31 03:02:09 dist-test-slave-j368 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828529, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:09.284153 8068 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:02:09.285275 8068 webserver.cc:492] Webserver started at http://127.6.22.65:35897/ using document root <none> and password file <none>
I20260131 03:02:09.285799 8068 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:09.285851 8068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:09.286000 8068 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:09.287699 8068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "894437dcf1ec4f40826976f686fab602"
format_stamp: "Formatted at 2026-01-31 03:02:09 on dist-test-slave-j368"
server_key: "72555e1e69d33a21396d0ad42b130098"
server_key_iv: "77caea9642f2efbb64af4b6aac35ac10"
server_key_version: "encryptionkey@0"
I20260131 03:02:09.288126 8068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "894437dcf1ec4f40826976f686fab602"
format_stamp: "Formatted at 2026-01-31 03:02:09 on dist-test-slave-j368"
server_key: "72555e1e69d33a21396d0ad42b130098"
server_key_iv: "77caea9642f2efbb64af4b6aac35ac10"
server_key_version: "encryptionkey@0"
I20260131 03:02:09.291620 8068 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260131 03:02:09.293919 8083 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:09.295028 8068 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:02:09.295146 8068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "894437dcf1ec4f40826976f686fab602"
format_stamp: "Formatted at 2026-01-31 03:02:09 on dist-test-slave-j368"
server_key: "72555e1e69d33a21396d0ad42b130098"
server_key_iv: "77caea9642f2efbb64af4b6aac35ac10"
server_key_version: "encryptionkey@0"
I20260131 03:02:09.295240 8068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:09.305944 8068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:09.309020 8068 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:09.309191 8068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:09.310599 8068 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:09.310684 8068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:09.310745 8068 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:09.310799 8068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:02:09 dist-test-slave-j368 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828529, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:09.324184 8068 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:35261
I20260131 03:02:09.324225 8198 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:35261 every 8 connection(s)
I20260131 03:02:09.325378 8068 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:02:09.326848 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.311359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:57793 (local address 127.6.22.126:45529)
0131 03:02:09.311522 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.311527 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.314322 (+ 2795us) server_negotiation.cc:408] Connection header received
0131 03:02:09.315438 (+ 1116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.315454 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.315535 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.315652 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:09.317363 (+ 1711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.318113 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.318889 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.319069 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.322773 (+ 3704us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:09.322809 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:09.322812 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:09.322843 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:09.324751 (+ 1908us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.325279 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.325285 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.325290 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.325351 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.325714 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.325721 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.325723 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.325938 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:09.326044 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.326607 (+ 563us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.326728 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260131 03:02:09.327399 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.311639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:45529 (local address 127.6.22.65:57793)
0131 03:02:09.314122 (+ 2483us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:09.314167 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:09.315168 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:09.315820 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:09.315832 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:09.316376 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:09.317209 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.317224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.318243 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.318249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:09.318758 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.318766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.319001 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.319985 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:09.320116 (+ 131us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:09.322559 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:09.324886 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.324891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.324899 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.325148 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.325492 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.325496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.325498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.325603 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.326107 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:09.326113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:09.326352 (+ 239us) client_negotiation.cc:769] Sending connection context
0131 03:02:09.326523 (+ 171us) client_negotiation.cc:240] Negotiation successful
0131 03:02:09.326748 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2287,"thread_start_us":2165,"threads_started":1}
I20260131 03:02:09.328099 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 8068
I20260131 03:02:09.328222 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:02:09.328473 6233 external_mini_cluster.cc:1468] Setting key 587f743443f9100b134720fe01392ab2
I20260131 03:02:09.328775 8199 heartbeater.cc:344] Connected to a master server at 127.6.22.126:45529
I20260131 03:02:09.328964 8199 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:09.329602 8199 heartbeater.cc:507] Master 127.6.22.126:45529 requested a full tablet report, sending...
I20260131 03:02:09.331194 7994 ts_manager.cc:194] Registered new tserver with Master: 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261)
I20260131 03:02:09.332788 7994 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:57793
I20260131 03:02:09.338259 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.331048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:37383 (local address 127.6.22.126:45529)
0131 03:02:09.331350 (+ 302us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.331355 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.331576 (+ 221us) server_negotiation.cc:408] Connection header received
0131 03:02:09.331723 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.331728 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.331785 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.331873 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:09.332558 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.333331 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.334201 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.334404 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.335270 (+ 866us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:09.335298 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:09.335301 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:09.335323 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:09.336788 (+ 1465us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.337188 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.337205 (+ 17us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.337210 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.337260 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.337522 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.337525 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.337527 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.337664 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:09.337769 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.338014 (+ 245us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.338115 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260131 03:02:09.338277 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.331009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:45529 (local address 127.6.22.65:37383)
0131 03:02:09.331492 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:09.331505 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:09.331620 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:09.331920 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:09.331924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:09.332127 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:09.332424 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.332432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.333516 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.333520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:09.334068 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.334077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.334217 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.334806 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:09.334819 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:09.335125 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:09.336916 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.336919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.336921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.337091 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.337375 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.337382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.337383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.337428 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.337760 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:09.337763 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:09.337850 (+ 87us) client_negotiation.cc:769] Sending connection context
0131 03:02:09.337954 (+ 104us) client_negotiation.cc:240] Negotiation successful
0131 03:02:09.338086 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:09.377517 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:45529
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:09.479642 8210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:09.479904 8210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:09.479996 8210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:09.483335 8210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:09.483441 8210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:09.483544 8210 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260131 03:02:09.483588 8210 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:02:09.483670 8210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:02:09.487984 8210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.22.126:45529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:09.489485 8210 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:09.490383 8210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:09.496855 8215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:09.496856 8216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:09.496855 8218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:09.497174 8210 server_base.cc:1060] running on GCE node
I20260131 03:02:09.497511 8210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:09.498081 8210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:09.499363 8210 hybrid_clock.cc:648] HybridClock initialized: now 1769828529499342 us; error 41 us; skew 500 ppm
Jan 31 03:02:09 dist-test-slave-j368 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828529, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:09.501962 8210 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:02:09.503002 8210 webserver.cc:492] Webserver started at http://127.6.22.66:45457/ using document root <none> and password file <none>
I20260131 03:02:09.503553 8210 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:09.503626 8210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:09.503826 8210 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:09.505538 8210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "f8b8b0afc91f4df189029455f588a7d4"
format_stamp: "Formatted at 2026-01-31 03:02:09 on dist-test-slave-j368"
server_key: "aa1d5188d001f7c5396480b3be9e0713"
server_key_iv: "7be4964d41d848f41df993eb60aec622"
server_key_version: "encryptionkey@0"
I20260131 03:02:09.505987 8210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "f8b8b0afc91f4df189029455f588a7d4"
format_stamp: "Formatted at 2026-01-31 03:02:09 on dist-test-slave-j368"
server_key: "aa1d5188d001f7c5396480b3be9e0713"
server_key_iv: "7be4964d41d848f41df993eb60aec622"
server_key_version: "encryptionkey@0"
I20260131 03:02:09.509306 8210 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:02:09.511559 8225 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:09.512579 8210 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:09.512719 8210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "f8b8b0afc91f4df189029455f588a7d4"
format_stamp: "Formatted at 2026-01-31 03:02:09 on dist-test-slave-j368"
server_key: "aa1d5188d001f7c5396480b3be9e0713"
server_key_iv: "7be4964d41d848f41df993eb60aec622"
server_key_version: "encryptionkey@0"
I20260131 03:02:09.512836 8210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:09.531155 8210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:09.534155 8210 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:09.534361 8210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:09.536000 8210 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:09.536077 8210 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:09.536150 8210 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:09.536211 8210 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:02:09 dist-test-slave-j368 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828529, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:09.547590 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.537756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:48519 (local address 127.6.22.126:45529)
0131 03:02:09.537904 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.537909 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.537926 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:02:09.538284 (+ 358us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.538288 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.538344 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.538454 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:09.539809 (+ 1355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.540438 (+ 629us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.541321 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.541501 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.543919 (+ 2418us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:09.543939 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:09.543942 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:09.543976 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:09.545460 (+ 1484us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.545975 (+ 515us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.545982 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.545987 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.546049 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.546460 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.546462 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.546464 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.546672 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:09.546779 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.547275 (+ 496us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.547424 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260131 03:02:09.547748 8210 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:41513
I20260131 03:02:09.548355 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.536763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:45529 (local address 127.6.22.66:48519)
0131 03:02:09.537184 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:09.537222 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:09.538053 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:09.538606 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:09.538613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:09.539015 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:09.539662 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.539675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.540554 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.540559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:09.541193 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.541205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.541427 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.542053 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:09.542104 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:09.543778 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:09.545575 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.545580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.545590 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.545850 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.546222 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.546229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.546233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.546365 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.546779 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:09.546787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:09.547043 (+ 256us) client_negotiation.cc:769] Sending connection context
0131 03:02:09.547240 (+ 197us) client_negotiation.cc:240] Negotiation successful
0131 03:02:09.547523 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":137,"threads_started":1}
I20260131 03:02:09.548836 8210 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:02:09.549564 8340 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:41513 every 8 connection(s)
I20260131 03:02:09.550709 8341 heartbeater.cc:344] Connected to a master server at 127.6.22.126:45529
I20260131 03:02:09.550843 8341 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:09.551280 8341 heartbeater.cc:507] Master 127.6.22.126:45529 requested a full tablet report, sending...
I20260131 03:02:09.552186 7994 ts_manager.cc:194] Registered new tserver with Master: f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513)
I20260131 03:02:09.552945 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 8210
I20260131 03:02:09.552953 7994 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:48519
I20260131 03:02:09.553052 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:02:09.553319 6233 external_mini_cluster.cc:1468] Setting key 80377ba2fa2bddef134eaa9994b42d39
I20260131 03:02:09.558610 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.551891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:40487 (local address 127.6.22.126:45529)
0131 03:02:09.552029 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.552033 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.552202 (+ 169us) server_negotiation.cc:408] Connection header received
0131 03:02:09.552367 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.552370 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.552409 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.552482 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:09.553179 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.553844 (+ 665us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.554468 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.554661 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.555446 (+ 785us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:09.555461 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:09.555464 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:09.555488 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:09.557043 (+ 1555us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.557496 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.557499 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.557501 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.557544 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.557797 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.557799 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.557800 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.557968 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:09.558047 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.558370 (+ 323us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.558503 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260131 03:02:09.558714 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.551831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:45529 (local address 127.6.22.66:40487)
0131 03:02:09.552122 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:09.552137 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:09.552250 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:09.552521 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:09.552524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:09.552760 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:09.553022 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.553031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.553935 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.553940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:09.554352 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.554360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.554493 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.555040 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:09.555052 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:09.555328 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:09.557154 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.557161 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.557166 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.557392 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.557638 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.557644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.557649 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.557711 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.558065 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:09.558071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:09.558171 (+ 100us) client_negotiation.cc:769] Sending connection context
0131 03:02:09.558291 (+ 120us) client_negotiation.cc:240] Negotiation successful
0131 03:02:09.558434 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:09.597218 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:45529
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:09.694828 8352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:09.695029 8352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:09.695137 8352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:09.698012 8352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:09.698089 8352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:09.698180 8352 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260131 03:02:09.698222 8352 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:02:09.698282 8352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:02:09.702028 8352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.22.126:45529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:09.702894 8352 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:09.703668 8352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:09.709250 8358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:09.709242 8360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:09.709228 8357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:09.709533 8352 server_base.cc:1060] running on GCE node
I20260131 03:02:09.709793 8352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:09.710281 8352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:09.711436 8352 hybrid_clock.cc:648] HybridClock initialized: now 1769828529711420 us; error 28 us; skew 500 ppm
Jan 31 03:02:09 dist-test-slave-j368 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828529, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:09.713694 8352 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:02:09.714545 8352 webserver.cc:492] Webserver started at http://127.6.22.67:44809/ using document root <none> and password file <none>
I20260131 03:02:09.715029 8352 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:09.715091 8352 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:09.715278 8352 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:09.716945 8352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "519e1d3f6c1645a3964028dd4efd4129"
format_stamp: "Formatted at 2026-01-31 03:02:09 on dist-test-slave-j368"
server_key: "9c0e61591b46df97f5360a8440dfffd3"
server_key_iv: "cacb6b18fa76040acaa407444848a53c"
server_key_version: "encryptionkey@0"
I20260131 03:02:09.717430 8352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "519e1d3f6c1645a3964028dd4efd4129"
format_stamp: "Formatted at 2026-01-31 03:02:09 on dist-test-slave-j368"
server_key: "9c0e61591b46df97f5360a8440dfffd3"
server_key_iv: "cacb6b18fa76040acaa407444848a53c"
server_key_version: "encryptionkey@0"
I20260131 03:02:09.720283 8352 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:02:09.722227 8367 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:09.723141 8352 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:02:09.723253 8352 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "519e1d3f6c1645a3964028dd4efd4129"
format_stamp: "Formatted at 2026-01-31 03:02:09 on dist-test-slave-j368"
server_key: "9c0e61591b46df97f5360a8440dfffd3"
server_key_iv: "cacb6b18fa76040acaa407444848a53c"
server_key_version: "encryptionkey@0"
I20260131 03:02:09.723356 8352 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:09.733122 8352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:09.735149 8352 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:09.735325 8352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:09.736589 8352 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:09.736654 8352 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:09.736734 8352 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:09.736778 8352 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:02:09 dist-test-slave-j368 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828529, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:09.746774 8352 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:40073
I20260131 03:02:09.746796 8482 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:40073 every 8 connection(s)
I20260131 03:02:09.747577 8352 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:02:09.747805 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.737181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:39305 (local address 127.6.22.126:45529)
0131 03:02:09.737318 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.737322 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.737970 (+ 648us) server_negotiation.cc:408] Connection header received
0131 03:02:09.738878 (+ 908us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.738880 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.738956 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.739047 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:09.740710 (+ 1663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.741505 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.742315 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.742491 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.744406 (+ 1915us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:09.744428 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:09.744431 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:09.744464 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:09.746038 (+ 1574us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.746438 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.746441 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.746443 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.746496 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.746799 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.746801 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.746803 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.746975 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:09.747075 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.747539 (+ 464us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.747656 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260131 03:02:09.748466 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.737403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:45529 (local address 127.6.22.67:39305)
0131 03:02:09.737816 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:09.737860 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:09.738690 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:09.739231 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:09.739240 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:09.739761 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:09.740522 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.740538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.741618 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.741623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:09.742201 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.742211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.742413 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.742957 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:09.742984 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:09.744276 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:09.746147 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.746151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.746158 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.746341 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.746586 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.746590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.746592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.746691 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.747073 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:09.747079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:09.747295 (+ 216us) client_negotiation.cc:769] Sending connection context
0131 03:02:09.747500 (+ 205us) client_negotiation.cc:240] Negotiation successful
0131 03:02:09.747728 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":90,"threads_started":1}
I20260131 03:02:09.749497 8483 heartbeater.cc:344] Connected to a master server at 127.6.22.126:45529
I20260131 03:02:09.749651 8483 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:09.750221 8483 heartbeater.cc:507] Master 127.6.22.126:45529 requested a full tablet report, sending...
I20260131 03:02:09.751152 7993 ts_manager.cc:194] Registered new tserver with Master: 519e1d3f6c1645a3964028dd4efd4129 (127.6.22.67:40073)
I20260131 03:02:09.751806 7993 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:39305
I20260131 03:02:09.752281 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 8352
I20260131 03:02:09.752396 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:02:09.752627 6233 external_mini_cluster.cc:1468] Setting key b6244b73316cf5bddf1c20ae6af5d5f9
I20260131 03:02:09.754935 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:02:09.758435 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.751537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:38775 (local address 127.6.22.126:45529)
0131 03:02:09.751636 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.751640 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.752898 (+ 1258us) server_negotiation.cc:408] Connection header received
0131 03:02:09.752990 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.752993 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.753031 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.753115 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:09.753681 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.754185 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.754826 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.754987 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.755643 (+ 656us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:09.755661 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:09.755664 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:09.755690 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:09.757148 (+ 1458us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.757548 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.757550 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.757551 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.757586 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.757791 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.757793 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.757794 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.757894 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:09.757985 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.758191 (+ 206us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.758302 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260131 03:02:09.758445 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.751471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:45529 (local address 127.6.22.67:38775)
0131 03:02:09.752810 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:09.752823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:09.752911 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:09.753148 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:09.753152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:09.753354 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:09.753591 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.753600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.754280 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.754285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:09.754727 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.754735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.754836 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.755270 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:09.755281 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:09.755530 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:09.757255 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.757260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.757263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.757456 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.757659 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.757663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.757666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.757718 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.757979 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:09.757982 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:09.758055 (+ 73us) client_negotiation.cc:769] Sending connection context
0131 03:02:09.758149 (+ 94us) client_negotiation.cc:240] Negotiation successful
0131 03:02:09.758252 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"thread_start_us":78,"threads_started":1}
Jan 31 03:02:09 dist-test-slave-j368 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 31 03:02:09 dist-test-slave-j368 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:09.776607 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.770538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.6.22.126:45529)
0131 03:02:09.770653 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.770656 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.770668 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:02:09.770699 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.770701 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.770736 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.770804 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:09.771551 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.771990 (+ 439us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.772585 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.772739 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.774398 (+ 1659us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:09.774411 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:09.774413 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:09.774432 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:09.775500 (+ 1068us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.775822 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.775825 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.775826 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.775862 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.776036 (+ 174us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.776038 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.776039 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.776153 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:09.776271 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.776425 (+ 154us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.776511 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:02:09.778779 7993 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:02:09.780980 7993 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:02:09.792626 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.787967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:40073 (local address 127.0.0.1:51224)
0131 03:02:09.788733 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:09.788744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:09.788817 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:09.789509 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:09.789513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:09.789531 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:09.789779 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.789786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.791217 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.791220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:09.792093 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.792103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.792230 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.792283 (+ 53us) client_negotiation.cc:769] Sending connection context
0131 03:02:09.792397 (+ 114us) client_negotiation.cc:240] Negotiation successful
0131 03:02:09.792478 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":90,"threads_started":1}
I20260131 03:02:09.792635 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.787856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:41513 (local address 127.0.0.1:56944)
0131 03:02:09.788142 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:09.788175 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:09.788302 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:09.789650 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:09.789653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:09.789669 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:09.789894 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.789901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.791248 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.791251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:09.792123 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.792132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.792252 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.792294 (+ 42us) client_negotiation.cc:769] Sending connection context
0131 03:02:09.792391 (+ 97us) client_negotiation.cc:240] Negotiation successful
0131 03:02:09.792490 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260131 03:02:09.792635 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.788504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:35261 (local address 127.0.0.1:49222)
0131 03:02:09.789710 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:09.789720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:09.789799 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:09.790208 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:09.790211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:09.790228 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:09.790443 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.790450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.791468 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.791471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:09.792309 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.792316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.792419 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.792431 (+ 12us) client_negotiation.cc:769] Sending connection context
0131 03:02:09.792479 (+ 48us) client_negotiation.cc:240] Negotiation successful
0131 03:02:09.792517 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"thread_start_us":77,"threads_started":1}
I20260131 03:02:09.793563 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.788796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.6.22.65:35261)
0131 03:02:09.789095 (+ 299us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.789101 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.789801 (+ 700us) server_negotiation.cc:408] Connection header received
0131 03:02:09.789903 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.789925 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.790077 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.790209 (+ 132us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:09.790556 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.791349 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.792460 (+ 1111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.793014 (+ 554us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.793119 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.793181 (+ 62us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.793272 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
I20260131 03:02:09.793605 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.788496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.6.22.66:41513)
0131 03:02:09.788909 (+ 413us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.788915 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.788938 (+ 23us) server_negotiation.cc:408] Connection header received
0131 03:02:09.789186 (+ 248us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.789192 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.789358 (+ 166us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.789497 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:09.790041 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.791150 (+ 1109us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.792747 (+ 1597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.793251 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.793341 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.793401 (+ 60us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.793498 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":106,"threads_started":1}
I20260131 03:02:09.793601 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.788371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.6.22.67:40073)
0131 03:02:09.789125 (+ 754us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.789137 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.789153 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:02:09.789215 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.789218 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.789379 (+ 161us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.789506 (+ 127us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:09.789900 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.791071 (+ 1171us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.792747 (+ 1676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.793263 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.793342 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.793401 (+ 59us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.793486 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":98,"threads_started":1}
I20260131 03:02:09.795845 8417 tablet_service.cc:1505] Processing CreateTablet for tablet d7721c68cc0c447a91c3cb45253d30ec (DEFAULT_TABLE table=test-table [id=97035d93b8f94172b0ae72b522a00193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:09.795950 8275 tablet_service.cc:1505] Processing CreateTablet for tablet d7721c68cc0c447a91c3cb45253d30ec (DEFAULT_TABLE table=test-table [id=97035d93b8f94172b0ae72b522a00193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:09.796252 8133 tablet_service.cc:1505] Processing CreateTablet for tablet d7721c68cc0c447a91c3cb45253d30ec (DEFAULT_TABLE table=test-table [id=97035d93b8f94172b0ae72b522a00193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:09.796787 8417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7721c68cc0c447a91c3cb45253d30ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:09.796808 8275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7721c68cc0c447a91c3cb45253d30ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:09.796937 8133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7721c68cc0c447a91c3cb45253d30ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:09.801585 8506 tablet_bootstrap.cc:492] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129: Bootstrap starting.
I20260131 03:02:09.802543 8508 tablet_bootstrap.cc:492] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4: Bootstrap starting.
I20260131 03:02:09.803349 8506 tablet_bootstrap.cc:654] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:09.803747 8507 tablet_bootstrap.cc:492] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602: Bootstrap starting.
I20260131 03:02:09.804121 8508 tablet_bootstrap.cc:654] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:09.804462 8506 log.cc:826] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:09.805214 8508 log.cc:826] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:09.805604 8507 tablet_bootstrap.cc:654] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:09.806077 8506 tablet_bootstrap.cc:492] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129: No bootstrap required, opened a new log
I20260131 03:02:09.806250 8506 ts_tablet_manager.cc:1403] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260131 03:02:09.806528 8507 log.cc:826] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:09.806732 8508 tablet_bootstrap.cc:492] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4: No bootstrap required, opened a new log
I20260131 03:02:09.806887 8508 ts_tablet_manager.cc:1403] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260131 03:02:09.808087 8507 tablet_bootstrap.cc:492] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602: No bootstrap required, opened a new log
I20260131 03:02:09.808249 8507 ts_tablet_manager.cc:1403] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260131 03:02:09.809027 8506 raft_consensus.cc:359] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } }
I20260131 03:02:09.809100 8508 raft_consensus.cc:359] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } }
I20260131 03:02:09.809216 8506 raft_consensus.cc:385] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:09.809258 8508 raft_consensus.cc:385] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:09.809258 8506 raft_consensus.cc:740] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519e1d3f6c1645a3964028dd4efd4129, State: Initialized, Role: FOLLOWER
I20260131 03:02:09.809302 8508 raft_consensus.cc:740] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b8b0afc91f4df189029455f588a7d4, State: Initialized, Role: FOLLOWER
I20260131 03:02:09.809634 8508 consensus_queue.cc:260] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } }
I20260131 03:02:09.809615 8506 consensus_queue.cc:260] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } }
I20260131 03:02:09.810181 8341 heartbeater.cc:499] Master 127.6.22.126:45529 was elected leader, sending a full tablet report...
I20260131 03:02:09.810386 8508 ts_tablet_manager.cc:1434] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260131 03:02:09.810390 8506 ts_tablet_manager.cc:1434] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260131 03:02:09.810696 8483 heartbeater.cc:499] Master 127.6.22.126:45529 was elected leader, sending a full tablet report...
I20260131 03:02:09.810907 8507 raft_consensus.cc:359] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } }
I20260131 03:02:09.811101 8507 raft_consensus.cc:385] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:09.811159 8507 raft_consensus.cc:740] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894437dcf1ec4f40826976f686fab602, State: Initialized, Role: FOLLOWER
I20260131 03:02:09.811503 8507 consensus_queue.cc:260] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } }
I20260131 03:02:09.812050 8199 heartbeater.cc:499] Master 127.6.22.126:45529 was elected leader, sending a full tablet report...
I20260131 03:02:09.812222 8507 ts_tablet_manager.cc:1434] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260131 03:02:09.826443 8200 tablet.cc:2378] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:02:09.865821 8512 raft_consensus.cc:493] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:02:09.865959 8512 raft_consensus.cc:515] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } }
I20260131 03:02:09.866886 8512 leader_election.cc:290] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 519e1d3f6c1645a3964028dd4efd4129 (127.6.22.67:40073), 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261)
I20260131 03:02:09.869588 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.867092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:40073 (local address 127.6.22.66:58907)
0131 03:02:09.867221 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:09.867237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:09.867316 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:09.867598 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:09.867601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:09.867619 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:09.867821 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.867828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.868566 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.868572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:09.869188 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.869197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.869322 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.869344 (+ 22us) client_negotiation.cc:769] Sending connection context
0131 03:02:09.869396 (+ 52us) client_negotiation.cc:240] Negotiation successful
0131 03:02:09.869469 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260131 03:02:09.870092 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.867199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:58907 (local address 127.6.22.67:40073)
0131 03:02:09.867326 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.867331 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.867376 (+ 45us) server_negotiation.cc:408] Connection header received
0131 03:02:09.867423 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.867426 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.867480 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.867582 (+ 102us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:09.867943 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.868443 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.869308 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.869820 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.869886 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.869930 (+ 44us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.869984 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260131 03:02:09.870117 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.867333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:35261 (local address 127.6.22.66:42783)
0131 03:02:09.867764 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:09.867776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:09.867905 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:09.868144 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:09.868147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:09.868174 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:09.868405 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.868417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.869100 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.869103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:09.869767 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.869776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.869929 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.869945 (+ 16us) client_negotiation.cc:769] Sending connection context
0131 03:02:09.869983 (+ 38us) client_negotiation.cc:240] Negotiation successful
0131 03:02:09.870023 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":133,"threads_started":1}
I20260131 03:02:09.870644 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.867427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:42783 (local address 127.6.22.65:35261)
0131 03:02:09.867568 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.867575 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.867860 (+ 285us) server_negotiation.cc:408] Connection header received
0131 03:02:09.868000 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.868004 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.868049 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.868133 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:09.868520 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.868993 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.869916 (+ 923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.870385 (+ 469us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.870434 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.870488 (+ 54us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.870565 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260131 03:02:09.870662 8435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7721c68cc0c447a91c3cb45253d30ec" candidate_uuid: "f8b8b0afc91f4df189029455f588a7d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519e1d3f6c1645a3964028dd4efd4129" is_pre_election: true
I20260131 03:02:09.870966 8435 raft_consensus.cc:2468] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8b8b0afc91f4df189029455f588a7d4 in term 0.
I20260131 03:02:09.871136 8153 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7721c68cc0c447a91c3cb45253d30ec" candidate_uuid: "f8b8b0afc91f4df189029455f588a7d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894437dcf1ec4f40826976f686fab602" is_pre_election: true
I20260131 03:02:09.871387 8153 raft_consensus.cc:2468] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8b8b0afc91f4df189029455f588a7d4 in term 0.
I20260131 03:02:09.871419 8226 leader_election.cc:304] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519e1d3f6c1645a3964028dd4efd4129, f8b8b0afc91f4df189029455f588a7d4; no voters:
I20260131 03:02:09.871625 8512 raft_consensus.cc:2804] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:02:09.871701 8512 raft_consensus.cc:493] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:02:09.871745 8512 raft_consensus.cc:3060] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:09.872611 8512 raft_consensus.cc:515] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } }
I20260131 03:02:09.872939 8512 leader_election.cc:290] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [CANDIDATE]: Term 1 election: Requested vote from peers 519e1d3f6c1645a3964028dd4efd4129 (127.6.22.67:40073), 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261)
I20260131 03:02:09.873250 8435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7721c68cc0c447a91c3cb45253d30ec" candidate_uuid: "f8b8b0afc91f4df189029455f588a7d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519e1d3f6c1645a3964028dd4efd4129"
I20260131 03:02:09.873298 8153 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7721c68cc0c447a91c3cb45253d30ec" candidate_uuid: "f8b8b0afc91f4df189029455f588a7d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894437dcf1ec4f40826976f686fab602"
I20260131 03:02:09.873337 8435 raft_consensus.cc:3060] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:09.873433 8153 raft_consensus.cc:3060] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:09.874339 8153 raft_consensus.cc:2468] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8b8b0afc91f4df189029455f588a7d4 in term 1.
I20260131 03:02:09.874369 8435 raft_consensus.cc:2468] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8b8b0afc91f4df189029455f588a7d4 in term 1.
I20260131 03:02:09.874646 8226 leader_election.cc:304] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519e1d3f6c1645a3964028dd4efd4129, f8b8b0afc91f4df189029455f588a7d4; no voters:
I20260131 03:02:09.874809 8512 raft_consensus.cc:2804] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:09.874997 8512 raft_consensus.cc:697] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 1 LEADER]: Becoming Leader. State: Replica: f8b8b0afc91f4df189029455f588a7d4, State: Running, Role: LEADER
I20260131 03:02:09.875300 8512 consensus_queue.cc:237] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } }
I20260131 03:02:09.877801 7992 catalog_manager.cc:5654] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66). New cstate: current_term: 1 leader_uuid: "f8b8b0afc91f4df189029455f588a7d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } health_report { overall_health: UNKNOWN } } }
Jan 31 03:02:09 dist-test-slave-j368 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828528, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:09.923393 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.914542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.6.22.126:45529)
0131 03:02:09.914684 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:09.914688 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:09.914896 (+ 208us) server_negotiation.cc:408] Connection header received
0131 03:02:09.914993 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:09.914996 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:09.915051 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:09.915127 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:09.915960 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.916622 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.917392 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.917524 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.920088 (+ 2564us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:09.920109 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:09.920111 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:09.920135 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:09.921647 (+ 1512us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.922152 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.922163 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.922165 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.922211 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:09.922521 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:09.922525 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:09.922526 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:09.922702 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:09.922791 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:09.923123 (+ 332us) server_negotiation.cc:300] Negotiation successful
0131 03:02:09.923236 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260131 03:02:09.923391 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:09.914451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:45529 (local address 127.0.0.1:53868)
0131 03:02:09.914799 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:09.914813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:09.914916 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:09.915208 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:09.915212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:09.915644 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:09.915837 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.915843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.916754 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:09.916757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:09.917274 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:09.917283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:09.917386 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:09.918011 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:09.918025 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:09.919923 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:09.921778 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.921788 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.921793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.922030 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.922332 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:09.922335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:09.922337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:09.922400 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:09.922834 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:09.922840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:09.922946 (+ 106us) client_negotiation.cc:769] Sending connection context
0131 03:02:09.923051 (+ 105us) client_negotiation.cc:240] Negotiation successful
0131 03:02:09.923172 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":142,"threads_started":1}
I20260131 03:02:09.928104 7992 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:53868:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260131 03:02:09.929252 7992 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:02:09.935496 8275 tablet_service.cc:1505] Processing CreateTablet for tablet 19f172e74eda4f3681b045ead15deb9b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1316eb0cd6bb4ff38bf53a746a7a6fde]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:02:09.935729 8417 tablet_service.cc:1505] Processing CreateTablet for tablet 19f172e74eda4f3681b045ead15deb9b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1316eb0cd6bb4ff38bf53a746a7a6fde]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:02:09.935423 8133 tablet_service.cc:1505] Processing CreateTablet for tablet 19f172e74eda4f3681b045ead15deb9b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1316eb0cd6bb4ff38bf53a746a7a6fde]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:02:09.936261 8417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f172e74eda4f3681b045ead15deb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:09.936262 8275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f172e74eda4f3681b045ead15deb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:09.936414 8133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f172e74eda4f3681b045ead15deb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:09.938737 8507 tablet_bootstrap.cc:492] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602: Bootstrap starting.
I20260131 03:02:09.939054 8508 tablet_bootstrap.cc:492] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4: Bootstrap starting.
I20260131 03:02:09.939441 8506 tablet_bootstrap.cc:492] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: Bootstrap starting.
I20260131 03:02:09.939779 8507 tablet_bootstrap.cc:654] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:09.940074 8508 tablet_bootstrap.cc:654] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:09.940550 8506 tablet_bootstrap.cc:654] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:09.941534 8508 tablet_bootstrap.cc:492] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4: No bootstrap required, opened a new log
I20260131 03:02:09.941618 8508 ts_tablet_manager.cc:1403] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260131 03:02:09.941699 8506 tablet_bootstrap.cc:492] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: No bootstrap required, opened a new log
I20260131 03:02:09.941699 8507 tablet_bootstrap.cc:492] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602: No bootstrap required, opened a new log
I20260131 03:02:09.941771 8506 ts_tablet_manager.cc:1403] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260131 03:02:09.941771 8507 ts_tablet_manager.cc:1403] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260131 03:02:09.942056 8508 raft_consensus.cc:359] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:09.942139 8508 raft_consensus.cc:385] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:09.942160 8508 raft_consensus.cc:740] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b8b0afc91f4df189029455f588a7d4, State: Initialized, Role: FOLLOWER
I20260131 03:02:09.942188 8507 raft_consensus.cc:359] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:09.942188 8506 raft_consensus.cc:359] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:09.942301 8506 raft_consensus.cc:385] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:09.942301 8507 raft_consensus.cc:385] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:09.942301 8508 consensus_queue.cc:260] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:09.942338 8506 raft_consensus.cc:740] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519e1d3f6c1645a3964028dd4efd4129, State: Initialized, Role: FOLLOWER
I20260131 03:02:09.942338 8507 raft_consensus.cc:740] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894437dcf1ec4f40826976f686fab602, State: Initialized, Role: FOLLOWER
I20260131 03:02:09.942452 8507 consensus_queue.cc:260] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:09.942452 8506 consensus_queue.cc:260] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:09.942517 8512 tablet_replica.cc:442] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } }
I20260131 03:02:09.942636 8512 tablet_replica.cc:445] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:09.942696 8506 ts_tablet_manager.cc:1434] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260131 03:02:09.942695 8507 ts_tablet_manager.cc:1434] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:09.942921 8514 tablet_replica.cc:442] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } }
I20260131 03:02:09.943022 8514 tablet_replica.cc:445] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:09.942569 8508 ts_tablet_manager.cc:1434] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260131 03:02:09.943442 8513 tablet_replica.cc:442] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } }
I20260131 03:02:09.943570 8513 tablet_replica.cc:445] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: This TxnStatusTablet replica's current role is: FOLLOWER
W20260131 03:02:09.998759 8484 tablet.cc:2378] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:02:10.050027 8342 tablet.cc:2378] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:02:10.174623 8513 raft_consensus.cc:493] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:02:10.174750 8513 raft_consensus.cc:515] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:10.175783 8513 leader_election.cc:290] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261), f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513)
I20260131 03:02:10.179167 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.175936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:35261 (local address 127.6.22.67:35773)
0131 03:02:10.176055 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:10.176072 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:10.176162 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:10.176441 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:10.176445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:10.176468 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:10.176769 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.176779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.177845 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.177848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:10.178787 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.178795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.178903 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.178922 (+ 19us) client_negotiation.cc:769] Sending connection context
0131 03:02:10.178971 (+ 49us) client_negotiation.cc:240] Negotiation successful
0131 03:02:10.179022 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"spinlock_wait_cycles":3328}
I20260131 03:02:10.179177 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.176021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:41513 (local address 127.6.22.67:53271)
0131 03:02:10.176397 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:10.176409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:10.176507 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:10.176791 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:10.176794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:10.176815 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:10.177009 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.177014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.177906 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.177909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:10.178787 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.178795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.178903 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.178922 (+ 19us) client_negotiation.cc:769] Sending connection context
0131 03:02:10.178971 (+ 49us) client_negotiation.cc:240] Negotiation successful
0131 03:02:10.179022 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":118,"threads_started":1}
I20260131 03:02:10.179561 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.176031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:35773 (local address 127.6.22.65:35261)
0131 03:02:10.176176 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:10.176182 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:10.176222 (+ 40us) server_negotiation.cc:408] Connection header received
0131 03:02:10.176281 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:10.176284 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:10.176334 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:10.176408 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:10.176978 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.177735 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.178905 (+ 1170us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.179328 (+ 423us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.179362 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:10.179407 (+ 45us) server_negotiation.cc:300] Negotiation successful
0131 03:02:10.179456 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260131 03:02:10.179562 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.176229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:53271 (local address 127.6.22.66:41513)
0131 03:02:10.176367 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:10.176372 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:10.176481 (+ 109us) server_negotiation.cc:408] Connection header received
0131 03:02:10.176625 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:10.176630 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:10.176701 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:10.176770 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:10.177107 (+ 337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.177806 (+ 699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.178905 (+ 1099us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.179327 (+ 422us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.179362 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:10.179406 (+ 44us) server_negotiation.cc:300] Negotiation successful
0131 03:02:10.179463 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260131 03:02:10.179998 8153 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19f172e74eda4f3681b045ead15deb9b" candidate_uuid: "519e1d3f6c1645a3964028dd4efd4129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894437dcf1ec4f40826976f686fab602" is_pre_election: true
I20260131 03:02:10.180119 8153 raft_consensus.cc:2468] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 519e1d3f6c1645a3964028dd4efd4129 in term 0.
I20260131 03:02:10.180111 8279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19f172e74eda4f3681b045ead15deb9b" candidate_uuid: "519e1d3f6c1645a3964028dd4efd4129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b8b0afc91f4df189029455f588a7d4" is_pre_election: true
I20260131 03:02:10.180366 8279 raft_consensus.cc:2468] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 519e1d3f6c1645a3964028dd4efd4129 in term 0.
I20260131 03:02:10.180557 8368 leader_election.cc:304] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519e1d3f6c1645a3964028dd4efd4129, 894437dcf1ec4f40826976f686fab602; no voters:
I20260131 03:02:10.180799 8513 raft_consensus.cc:2804] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:02:10.180863 8513 raft_consensus.cc:493] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:02:10.180900 8513 raft_consensus.cc:3060] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:10.181753 8513 raft_consensus.cc:515] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:10.182066 8513 leader_election.cc:290] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [CANDIDATE]: Term 1 election: Requested vote from peers 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261), f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513)
I20260131 03:02:10.182364 8153 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19f172e74eda4f3681b045ead15deb9b" candidate_uuid: "519e1d3f6c1645a3964028dd4efd4129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894437dcf1ec4f40826976f686fab602"
I20260131 03:02:10.182468 8153 raft_consensus.cc:3060] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:10.182461 8279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19f172e74eda4f3681b045ead15deb9b" candidate_uuid: "519e1d3f6c1645a3964028dd4efd4129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b8b0afc91f4df189029455f588a7d4"
I20260131 03:02:10.182608 8279 raft_consensus.cc:3060] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:10.183331 8153 raft_consensus.cc:2468] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 519e1d3f6c1645a3964028dd4efd4129 in term 1.
I20260131 03:02:10.183545 8279 raft_consensus.cc:2468] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 519e1d3f6c1645a3964028dd4efd4129 in term 1.
I20260131 03:02:10.183620 8368 leader_election.cc:304] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519e1d3f6c1645a3964028dd4efd4129, 894437dcf1ec4f40826976f686fab602; no voters:
I20260131 03:02:10.183795 8513 raft_consensus.cc:2804] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:10.184022 8513 raft_consensus.cc:697] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [term 1 LEADER]: Becoming Leader. State: Replica: 519e1d3f6c1645a3964028dd4efd4129, State: Running, Role: LEADER
I20260131 03:02:10.184319 8513 consensus_queue.cc:237] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:10.186213 8528 tablet_replica.cc:442] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: TxnStatusTablet state changed. Reason: New leader 519e1d3f6c1645a3964028dd4efd4129. Latest consensus state: current_term: 1 leader_uuid: "519e1d3f6c1645a3964028dd4efd4129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } }
I20260131 03:02:10.186301 8528 tablet_replica.cc:445] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:02:10.186546 8530 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260131 03:02:10.186782 8530 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260131 03:02:10.187377 7994 catalog_manager.cc:5654] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 reported cstate change: term changed from 0 to 1, leader changed from <none> to 519e1d3f6c1645a3964028dd4efd4129 (127.6.22.67). New cstate: current_term: 1 leader_uuid: "519e1d3f6c1645a3964028dd4efd4129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } health_report { overall_health: HEALTHY } } }
I20260131 03:02:10.234059 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.231346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51232 (local address 127.6.22.67:40073)
0131 03:02:10.231462 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:10.231465 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:10.231478 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:02:10.231616 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:10.231619 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:10.231664 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:10.231730 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:10.232081 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.232494 (+ 413us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.233243 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.233353 (+ 110us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.233393 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:10.233749 (+ 356us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:10.233825 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:10.233925 (+ 100us) server_negotiation.cc:300] Negotiation successful
0131 03:02:10.233966 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:02:10.234083 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.231234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:40073 (local address 127.0.0.1:51232)
0131 03:02:10.231378 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:10.231391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:10.231498 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:10.231757 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:10.231760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:10.231767 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:02:10.231987 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.231995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.232586 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.232589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:10.233147 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.233156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.233256 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.233293 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:02:10.233833 (+ 540us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:02:10.233845 (+ 12us) client_negotiation.cc:769] Sending connection context
0131 03:02:10.233909 (+ 64us) client_negotiation.cc:240] Negotiation successful
0131 03:02:10.233969 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260131 03:02:10.238420 8279 raft_consensus.cc:1275] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Refusing update from remote peer 519e1d3f6c1645a3964028dd4efd4129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:10.238469 8153 raft_consensus.cc:1275] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Refusing update from remote peer 519e1d3f6c1645a3964028dd4efd4129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:10.238943 8513 consensus_queue.cc:1048] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:10.239094 8528 consensus_queue.cc:1048] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:10.241788 8512 tablet_replica.cc:442] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4: TxnStatusTablet state changed. Reason: New leader 519e1d3f6c1645a3964028dd4efd4129. Latest consensus state: current_term: 1 leader_uuid: "519e1d3f6c1645a3964028dd4efd4129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } }
I20260131 03:02:10.241894 8512 tablet_replica.cc:445] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:10.242290 8514 tablet_replica.cc:442] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602: TxnStatusTablet state changed. Reason: New leader 519e1d3f6c1645a3964028dd4efd4129. Latest consensus state: current_term: 1 leader_uuid: "519e1d3f6c1645a3964028dd4efd4129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } }
I20260131 03:02:10.242378 8514 tablet_replica.cc:445] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:10.244913 8528 tablet_replica.cc:442] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "519e1d3f6c1645a3964028dd4efd4129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } }
I20260131 03:02:10.245003 8528 tablet_replica.cc:445] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:02:10.245111 8528 tablet_replica.cc:442] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "519e1d3f6c1645a3964028dd4efd4129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } }
I20260131 03:02:10.245173 8528 tablet_replica.cc:445] T 19f172e74eda4f3681b045ead15deb9b P 519e1d3f6c1645a3964028dd4efd4129: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:02:10.245595 8512 tablet_replica.cc:442] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "519e1d3f6c1645a3964028dd4efd4129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } }
I20260131 03:02:10.245685 8512 tablet_replica.cc:445] T 19f172e74eda4f3681b045ead15deb9b P f8b8b0afc91f4df189029455f588a7d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:10.246289 8514 tablet_replica.cc:442] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "519e1d3f6c1645a3964028dd4efd4129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } } }
I20260131 03:02:10.246308 8534 mvcc.cc:204] Tried to move back new op lower bound from 7249217659851149312 to 7249217659637485568. Current Snapshot: MvccSnapshot[applied={T|T < 7249217659851149312}]
I20260131 03:02:10.246369 8514 tablet_replica.cc:445] T 19f172e74eda4f3681b045ead15deb9b P 894437dcf1ec4f40826976f686fab602: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:02:10.247479 8535 mvcc.cc:204] Tried to move back new op lower bound from 7249217659851149312 to 7249217659637485568. Current Snapshot: MvccSnapshot[applied={T|T < 7249217659851149312}]
I20260131 03:02:10.254921 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.252111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.6.22.66:41513)
0131 03:02:10.252209 (+ 98us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:10.252213 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:10.252225 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:02:10.252341 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:10.252344 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:10.252396 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:10.252485 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:10.252793 (+ 308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.253425 (+ 632us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.254004 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.254165 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.254215 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:10.254629 (+ 414us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:10.254722 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:10.254775 (+ 53us) server_negotiation.cc:300] Negotiation successful
0131 03:02:10.254820 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:02:10.263279 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.260926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:39411 (local address 127.6.22.67:40073)
0131 03:02:10.261064 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:10.261068 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:10.261217 (+ 149us) server_negotiation.cc:408] Connection header received
0131 03:02:10.261366 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:10.261369 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:10.261413 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:10.261490 (+ 77us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:10.261796 (+ 306us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.262174 (+ 378us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.262722 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.262893 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.262928 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:10.262976 (+ 48us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:10.263049 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:10.263143 (+ 94us) server_negotiation.cc:300] Negotiation successful
0131 03:02:10.263179 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260131 03:02:10.263295 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.260851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:40073 (local address 127.6.22.66:39411)
0131 03:02:10.261119 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:10.261130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:10.261280 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:10.261496 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:10.261499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:10.261506 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:02:10.261703 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.261708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.262270 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.262274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:10.262622 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.262628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.262733 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.262766 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:02:10.263055 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:02:10.263062 (+ 7us) client_negotiation.cc:769] Sending connection context
0131 03:02:10.263133 (+ 71us) client_negotiation.cc:240] Negotiation successful
0131 03:02:10.263186 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
I20260131 03:02:10.269033 8153 raft_consensus.cc:1275] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Refusing update from remote peer f8b8b0afc91f4df189029455f588a7d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:10.269269 8435 raft_consensus.cc:1275] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Refusing update from remote peer f8b8b0afc91f4df189029455f588a7d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:10.269423 8516 consensus_queue.cc:1048] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:10.269645 8512 consensus_queue.cc:1048] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:10.289392 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.286400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:40345 (local address 127.6.22.66:41513)
0131 03:02:10.286508 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:10.286512 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:10.286825 (+ 313us) server_negotiation.cc:408] Connection header received
0131 03:02:10.286914 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:10.286917 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:10.286959 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:10.287027 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:10.287302 (+ 275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.287954 (+ 652us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.288627 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.288809 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.288973 (+ 164us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:10.289038 (+ 65us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:10.289098 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:10.289248 (+ 150us) server_negotiation.cc:300] Negotiation successful
0131 03:02:10.289293 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260131 03:02:10.289453 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.286315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:41513 (local address 127.6.22.67:40345)
0131 03:02:10.286749 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:10.286767 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:10.286843 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:10.287039 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:10.287041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:10.287046 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:02:10.287215 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.287219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.288051 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.288054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:10.288520 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.288531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.288633 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.288710 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:02:10.289121 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:02:10.289126 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:02:10.289277 (+ 151us) client_negotiation.cc:240] Negotiation successful
0131 03:02:10.289331 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":87,"threads_started":1}
I20260131 03:02:10.309104 7994 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:10.309257 7994 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:10.311287 7994 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480: Sending DeleteTablet for 3 replicas of tablet d7721c68cc0c447a91c3cb45253d30ec
I20260131 03:02:10.311822 8275 tablet_service.cc:1552] Processing DeleteTablet for tablet d7721c68cc0c447a91c3cb45253d30ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:10 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:56944
I20260131 03:02:10.311918 8133 tablet_service.cc:1552] Processing DeleteTablet for tablet d7721c68cc0c447a91c3cb45253d30ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:10 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:49222
I20260131 03:02:10.312280 8553 tablet_replica.cc:333] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4: stopping tablet replica
I20260131 03:02:10.312328 8417 tablet_service.cc:1552] Processing DeleteTablet for tablet d7721c68cc0c447a91c3cb45253d30ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:10 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:51224
I20260131 03:02:10.312402 7994 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260131 03:02:10.312737 8553 raft_consensus.cc:2243] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 1 LEADER]: Raft consensus shutting down.
W20260131 03:02:10.312788 7994 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:02:10.313129 8553 raft_consensus.cc:2272] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:10.313599 8555 tablet_replica.cc:333] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129: stopping tablet replica
I20260131 03:02:10.313791 8555 raft_consensus.cc:2243] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:10.313923 8555 raft_consensus.cc:2272] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:10.314945 8555 ts_tablet_manager.cc:1916] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:10.317929 8133 tablet_service.cc:1505] Processing CreateTablet for tablet 4a3fa9d5961b47e3acc2b067d3fb8280 (DEFAULT_TABLE table=test-table [id=ccef7ca2ab144382b55adef7b8e341c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:10.318145 8133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a3fa9d5961b47e3acc2b067d3fb8280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:10.318689 8555 ts_tablet_manager.cc:1929] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:02:10.318753 8555 log.cc:1199] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/d7721c68cc0c447a91c3cb45253d30ec
I20260131 03:02:10.318985 8555 ts_tablet_manager.cc:1950] T d7721c68cc0c447a91c3cb45253d30ec P 519e1d3f6c1645a3964028dd4efd4129: Deleting consensus metadata
I20260131 03:02:10.320423 8417 tablet_service.cc:1505] Processing CreateTablet for tablet 4a3fa9d5961b47e3acc2b067d3fb8280 (DEFAULT_TABLE table=test-table [id=ccef7ca2ab144382b55adef7b8e341c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:10.320622 7978 catalog_manager.cc:4985] TS 519e1d3f6c1645a3964028dd4efd4129 (127.6.22.67:40073): tablet d7721c68cc0c447a91c3cb45253d30ec (table test-table [id=97035d93b8f94172b0ae72b522a00193]) successfully deleted
I20260131 03:02:10.320691 8417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a3fa9d5961b47e3acc2b067d3fb8280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:10.320801 8554 tablet_replica.cc:333] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602: stopping tablet replica
I20260131 03:02:10.321004 8554 raft_consensus.cc:2243] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:10.321192 8554 raft_consensus.cc:2272] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:10.322178 8506 tablet_bootstrap.cc:492] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129: Bootstrap starting.
I20260131 03:02:10.322190 8554 ts_tablet_manager.cc:1916] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:10.322242 8507 tablet_bootstrap.cc:492] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602: Bootstrap starting.
I20260131 03:02:10.322625 8275 tablet_service.cc:1505] Processing CreateTablet for tablet 4a3fa9d5961b47e3acc2b067d3fb8280 (DEFAULT_TABLE table=test-table [id=ccef7ca2ab144382b55adef7b8e341c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:10.322826 8275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a3fa9d5961b47e3acc2b067d3fb8280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:10.323012 8506 tablet_bootstrap.cc:654] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:10.324693 8508 tablet_bootstrap.cc:492] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4: Bootstrap starting.
I20260131 03:02:10.325551 8508 tablet_bootstrap.cc:654] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:10.325650 8507 tablet_bootstrap.cc:654] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:10.326692 8507 tablet_bootstrap.cc:492] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602: No bootstrap required, opened a new log
I20260131 03:02:10.326771 8507 ts_tablet_manager.cc:1403] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260131 03:02:10.327236 8507 raft_consensus.cc:359] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:10.327350 8507 raft_consensus.cc:385] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:10.327384 8507 raft_consensus.cc:740] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894437dcf1ec4f40826976f686fab602, State: Initialized, Role: FOLLOWER
I20260131 03:02:10.327489 8507 consensus_queue.cc:260] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:10.327708 8507 ts_tablet_manager.cc:1434] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:10.329137 8553 ts_tablet_manager.cc:1916] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:10.331303 8553 ts_tablet_manager.cc:1929] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:02:10.331396 8553 log.cc:1199] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/d7721c68cc0c447a91c3cb45253d30ec
I20260131 03:02:10.331722 8553 ts_tablet_manager.cc:1950] T d7721c68cc0c447a91c3cb45253d30ec P f8b8b0afc91f4df189029455f588a7d4: Deleting consensus metadata
I20260131 03:02:10.332585 7981 catalog_manager.cc:4985] TS f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513): tablet d7721c68cc0c447a91c3cb45253d30ec (table test-table [id=97035d93b8f94172b0ae72b522a00193]) successfully deleted
I20260131 03:02:10.333081 8506 tablet_bootstrap.cc:492] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129: No bootstrap required, opened a new log
I20260131 03:02:10.333180 8506 ts_tablet_manager.cc:1403] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20260131 03:02:10.333534 8554 ts_tablet_manager.cc:1929] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:02:10.333604 8554 log.cc:1199] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/wal/wals/d7721c68cc0c447a91c3cb45253d30ec
I20260131 03:02:10.333709 8506 raft_consensus.cc:359] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:10.333806 8506 raft_consensus.cc:385] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:10.333839 8506 raft_consensus.cc:740] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519e1d3f6c1645a3964028dd4efd4129, State: Initialized, Role: FOLLOWER
I20260131 03:02:10.333849 8554 ts_tablet_manager.cc:1950] T d7721c68cc0c447a91c3cb45253d30ec P 894437dcf1ec4f40826976f686fab602: Deleting consensus metadata
I20260131 03:02:10.333956 8506 consensus_queue.cc:260] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:10.334179 8506 ts_tablet_manager.cc:1434] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260131 03:02:10.334681 7978 catalog_manager.cc:4985] TS 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261): tablet d7721c68cc0c447a91c3cb45253d30ec (table test-table [id=97035d93b8f94172b0ae72b522a00193]) successfully deleted
I20260131 03:02:10.341341 8508 tablet_bootstrap.cc:492] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4: No bootstrap required, opened a new log
I20260131 03:02:10.341406 8508 ts_tablet_manager.cc:1403] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260131 03:02:10.341773 8508 raft_consensus.cc:359] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:10.341850 8508 raft_consensus.cc:385] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:10.341874 8508 raft_consensus.cc:740] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b8b0afc91f4df189029455f588a7d4, State: Initialized, Role: FOLLOWER
I20260131 03:02:10.341964 8508 consensus_queue.cc:260] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:10.342201 8508 ts_tablet_manager.cc:1434] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:10.556152 8514 raft_consensus.cc:493] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:02:10.556284 8514 raft_consensus.cc:515] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:10.557301 8514 leader_election.cc:290] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513), 519e1d3f6c1645a3964028dd4efd4129 (127.6.22.67:40073)
I20260131 03:02:10.560303 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.557487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:41513 (local address 127.6.22.65:60579)
0131 03:02:10.557772 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:10.557785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:10.557872 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:10.558072 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:10.558074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:10.558088 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:10.558331 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.558337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.559133 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.559136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:10.559869 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.559877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.559994 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.560014 (+ 20us) client_negotiation.cc:769] Sending connection context
0131 03:02:10.560083 (+ 69us) client_negotiation.cc:240] Negotiation successful
0131 03:02:10.560145 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260131 03:02:10.560742 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.557601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:45323 (local address 127.6.22.67:40073)
0131 03:02:10.557708 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:10.557713 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:10.557886 (+ 173us) server_negotiation.cc:408] Connection header received
0131 03:02:10.558057 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:10.558061 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:10.558105 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:10.558163 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:10.558521 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.558920 (+ 399us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.559985 (+ 1065us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.560462 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.560500 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:10.560550 (+ 50us) server_negotiation.cc:300] Negotiation successful
0131 03:02:10.560600 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260131 03:02:10.560943 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.557788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:60579 (local address 127.6.22.66:41513)
0131 03:02:10.557898 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:10.557903 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:10.557916 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:02:10.557954 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:10.557957 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:10.558001 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:10.558058 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:10.558438 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.559039 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.560220 (+ 1181us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.560621 (+ 401us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.560698 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:10.560751 (+ 53us) server_negotiation.cc:300] Negotiation successful
0131 03:02:10.560829 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260131 03:02:10.561312 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.557487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:40073 (local address 127.6.22.65:45323)
0131 03:02:10.557813 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:10.557826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:10.557953 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:10.558173 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:10.558176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:10.558188 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:10.558369 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.558376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.559036 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.559040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:10.559869 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.559877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.559993 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.560014 (+ 21us) client_negotiation.cc:769] Sending connection context
0131 03:02:10.560083 (+ 69us) client_negotiation.cc:240] Negotiation successful
0131 03:02:10.560145 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":9,"thread_start_us":71,"threads_started":1}
I20260131 03:02:10.561443 8279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a3fa9d5961b47e3acc2b067d3fb8280" candidate_uuid: "894437dcf1ec4f40826976f686fab602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b8b0afc91f4df189029455f588a7d4" is_pre_election: true
I20260131 03:02:10.561558 8279 raft_consensus.cc:2468] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 894437dcf1ec4f40826976f686fab602 in term 0.
I20260131 03:02:10.561895 8435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a3fa9d5961b47e3acc2b067d3fb8280" candidate_uuid: "894437dcf1ec4f40826976f686fab602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519e1d3f6c1645a3964028dd4efd4129" is_pre_election: true
I20260131 03:02:10.562021 8435 raft_consensus.cc:2468] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 894437dcf1ec4f40826976f686fab602 in term 0.
I20260131 03:02:10.562008 8087 leader_election.cc:304] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 894437dcf1ec4f40826976f686fab602, f8b8b0afc91f4df189029455f588a7d4; no voters:
I20260131 03:02:10.562255 8514 raft_consensus.cc:2804] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:02:10.562318 8514 raft_consensus.cc:493] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:02:10.562384 8514 raft_consensus.cc:3060] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:10.563198 8514 raft_consensus.cc:515] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:10.563584 8514 leader_election.cc:290] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [CANDIDATE]: Term 1 election: Requested vote from peers f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513), 519e1d3f6c1645a3964028dd4efd4129 (127.6.22.67:40073)
I20260131 03:02:10.563838 8279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a3fa9d5961b47e3acc2b067d3fb8280" candidate_uuid: "894437dcf1ec4f40826976f686fab602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b8b0afc91f4df189029455f588a7d4"
I20260131 03:02:10.563913 8435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a3fa9d5961b47e3acc2b067d3fb8280" candidate_uuid: "894437dcf1ec4f40826976f686fab602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519e1d3f6c1645a3964028dd4efd4129"
I20260131 03:02:10.563958 8279 raft_consensus.cc:3060] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:10.564015 8435 raft_consensus.cc:3060] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:10.564754 8279 raft_consensus.cc:2468] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 894437dcf1ec4f40826976f686fab602 in term 1.
I20260131 03:02:10.564956 8435 raft_consensus.cc:2468] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 894437dcf1ec4f40826976f686fab602 in term 1.
I20260131 03:02:10.565007 8087 leader_election.cc:304] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 894437dcf1ec4f40826976f686fab602, f8b8b0afc91f4df189029455f588a7d4; no voters:
I20260131 03:02:10.565164 8514 raft_consensus.cc:2804] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:10.565335 8514 raft_consensus.cc:697] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 1 LEADER]: Becoming Leader. State: Replica: 894437dcf1ec4f40826976f686fab602, State: Running, Role: LEADER
I20260131 03:02:10.565531 8514 consensus_queue.cc:237] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:10.567456 7994 catalog_manager.cc:5654] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 894437dcf1ec4f40826976f686fab602 (127.6.22.65). New cstate: current_term: 1 leader_uuid: "894437dcf1ec4f40826976f686fab602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } health_report { overall_health: UNKNOWN } } }
I20260131 03:02:10.617074 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.614527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.6.22.65:35261)
0131 03:02:10.614634 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:10.614638 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:10.614714 (+ 76us) server_negotiation.cc:408] Connection header received
0131 03:02:10.614829 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:10.614831 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:10.614874 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:10.614940 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:10.615209 (+ 269us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.615619 (+ 410us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.616123 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.616229 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.616277 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:10.616697 (+ 420us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:10.616773 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:10.616874 (+ 101us) server_negotiation.cc:300] Negotiation successful
0131 03:02:10.616910 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260131 03:02:10.624832 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.622788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:40073 (local address 127.6.22.65:59059)
0131 03:02:10.623007 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:10.623018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:10.623087 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:10.623286 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:10.623288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:10.623293 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:02:10.623465 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.623468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.623974 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.623976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:10.624321 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.624326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.624395 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.624420 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:02:10.624665 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:02:10.624668 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:02:10.624709 (+ 41us) client_negotiation.cc:240] Negotiation successful
0131 03:02:10.624750 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260131 03:02:10.625041 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.622831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:59059 (local address 127.6.22.67:40073)
0131 03:02:10.622941 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:10.622945 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:10.623073 (+ 128us) server_negotiation.cc:408] Connection header received
0131 03:02:10.623158 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:10.623160 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:10.623197 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:10.623261 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:10.623541 (+ 280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.623898 (+ 357us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.624398 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.624503 (+ 105us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.624532 (+ 29us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:10.624579 (+ 47us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:10.624630 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:10.624853 (+ 223us) server_negotiation.cc:300] Negotiation successful
0131 03:02:10.624922 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260131 03:02:10.630066 8435 raft_consensus.cc:1275] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Refusing update from remote peer 894437dcf1ec4f40826976f686fab602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:10.630108 8279 raft_consensus.cc:1275] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Refusing update from remote peer 894437dcf1ec4f40826976f686fab602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:10.630546 8514 consensus_queue.cc:1048] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:10.630712 8561 consensus_queue.cc:1048] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:10.645004 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.642812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:51033 (local address 127.6.22.65:35261)
0131 03:02:10.642892 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:10.642894 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:10.643003 (+ 109us) server_negotiation.cc:408] Connection header received
0131 03:02:10.643138 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:10.643155 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:10.643199 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:10.643264 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:10.643600 (+ 336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.643957 (+ 357us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.644463 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.644559 (+ 96us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.644585 (+ 26us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:10.644631 (+ 46us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:10.644725 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:10.644867 (+ 142us) server_negotiation.cc:300] Negotiation successful
0131 03:02:10.644899 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260131 03:02:10.645012 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:10.642739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:35261 (local address 127.6.22.67:51033)
0131 03:02:10.642929 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:10.642943 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:10.643031 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:10.643286 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:10.643289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:10.643297 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:02:10.643488 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.643497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.644036 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:10.644038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:10.644376 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:10.644382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:10.644467 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:10.644487 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:02:10.644764 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:02:10.644769 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:02:10.644845 (+ 76us) client_negotiation.cc:240] Negotiation successful
0131 03:02:10.644913 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
W20260131 03:02:10.645673 8535 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260131 03:02:12.210422 7994 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:12.210566 7994 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:12.213018 7994 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480: Sending DeleteTablet for 3 replicas of tablet 4a3fa9d5961b47e3acc2b067d3fb8280
I20260131 03:02:12.213629 8133 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a3fa9d5961b47e3acc2b067d3fb8280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:12 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:49222
I20260131 03:02:12.213682 8275 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a3fa9d5961b47e3acc2b067d3fb8280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:12 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:56944
I20260131 03:02:12.213759 8417 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a3fa9d5961b47e3acc2b067d3fb8280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:12 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:51224
I20260131 03:02:12.214013 8592 tablet_replica.cc:333] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4: stopping tablet replica
I20260131 03:02:12.214169 8592 raft_consensus.cc:2243] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:12.214314 8592 raft_consensus.cc:2272] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:12.214591 8593 tablet_replica.cc:333] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129: stopping tablet replica
I20260131 03:02:12.214778 8593 raft_consensus.cc:2243] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:12.214937 8593 raft_consensus.cc:2272] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:12.215044 8592 ts_tablet_manager.cc:1916] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:12.216229 8593 ts_tablet_manager.cc:1916] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:12.217206 8591 tablet_replica.cc:333] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602: stopping tablet replica
I20260131 03:02:12.217350 8591 raft_consensus.cc:2243] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:02:12.217694 8591 raft_consensus.cc:2272] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:12.218250 8592 ts_tablet_manager.cc:1929] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260131 03:02:12.218346 8592 log.cc:1199] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/4a3fa9d5961b47e3acc2b067d3fb8280
I20260131 03:02:12.218377 8591 ts_tablet_manager.cc:1916] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:12.218670 8592 ts_tablet_manager.cc:1950] T 4a3fa9d5961b47e3acc2b067d3fb8280 P f8b8b0afc91f4df189029455f588a7d4: Deleting consensus metadata
I20260131 03:02:12.219388 7981 catalog_manager.cc:4985] TS f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513): tablet 4a3fa9d5961b47e3acc2b067d3fb8280 (table test-table [id=ccef7ca2ab144382b55adef7b8e341c7]) successfully deleted
I20260131 03:02:12.219580 8593 ts_tablet_manager.cc:1929] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260131 03:02:12.219676 8593 log.cc:1199] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/4a3fa9d5961b47e3acc2b067d3fb8280
I20260131 03:02:12.220005 8593 ts_tablet_manager.cc:1950] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 519e1d3f6c1645a3964028dd4efd4129: Deleting consensus metadata
I20260131 03:02:12.220780 7978 catalog_manager.cc:4985] TS 519e1d3f6c1645a3964028dd4efd4129 (127.6.22.67:40073): tablet 4a3fa9d5961b47e3acc2b067d3fb8280 (table test-table [id=ccef7ca2ab144382b55adef7b8e341c7]) successfully deleted
I20260131 03:02:12.221077 8591 ts_tablet_manager.cc:1929] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260131 03:02:12.221177 8591 log.cc:1199] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/wal/wals/4a3fa9d5961b47e3acc2b067d3fb8280
I20260131 03:02:12.221508 8591 ts_tablet_manager.cc:1950] T 4a3fa9d5961b47e3acc2b067d3fb8280 P 894437dcf1ec4f40826976f686fab602: Deleting consensus metadata
I20260131 03:02:12.222178 7978 catalog_manager.cc:4985] TS 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261): tablet 4a3fa9d5961b47e3acc2b067d3fb8280 (table test-table [id=ccef7ca2ab144382b55adef7b8e341c7]) successfully deleted
Jan 31 03:02:12 dist-test-slave-j368 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828532, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 31 03:02:12 dist-test-slave-j368 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828532, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:12.242633 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:12.231552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.6.22.126:45529)
0131 03:02:12.231832 (+ 280us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:12.231836 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:12.231852 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:02:12.231914 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:12.231916 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:12.231961 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:12.232043 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:12.232847 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:12.233383 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:12.233971 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:12.234099 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:12.239440 (+ 5341us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:12.239469 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:12.239472 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:12.239497 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:12.241065 (+ 1568us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:12.241506 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:12.241510 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:12.241512 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:12.241581 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:12.241817 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:12.241820 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:12.241822 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:12.241976 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:12.242111 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:12.242338 (+ 227us) server_negotiation.cc:300] Negotiation successful
0131 03:02:12.242461 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260131 03:02:12.244990 7994 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:02:12.245371 7994 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:02:12.249866 8133 tablet_service.cc:1505] Processing CreateTablet for tablet 8d691317f7f345fa8b9a7a271d980eba (DEFAULT_TABLE table=test-table [id=b84096c0e8dd47e18e587320add3d4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:12.249969 8275 tablet_service.cc:1505] Processing CreateTablet for tablet 8d691317f7f345fa8b9a7a271d980eba (DEFAULT_TABLE table=test-table [id=b84096c0e8dd47e18e587320add3d4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:12.250190 8133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d691317f7f345fa8b9a7a271d980eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:12.250206 8275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d691317f7f345fa8b9a7a271d980eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:12.250172 8417 tablet_service.cc:1505] Processing CreateTablet for tablet 8d691317f7f345fa8b9a7a271d980eba (DEFAULT_TABLE table=test-table [id=b84096c0e8dd47e18e587320add3d4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:12.250439 8417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d691317f7f345fa8b9a7a271d980eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:12.252305 8602 tablet_bootstrap.cc:492] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129: Bootstrap starting.
I20260131 03:02:12.252862 8603 tablet_bootstrap.cc:492] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4: Bootstrap starting.
I20260131 03:02:12.253263 8602 tablet_bootstrap.cc:654] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:12.253809 8603 tablet_bootstrap.cc:654] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:12.254439 8604 tablet_bootstrap.cc:492] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602: Bootstrap starting.
I20260131 03:02:12.254480 8602 tablet_bootstrap.cc:492] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129: No bootstrap required, opened a new log
I20260131 03:02:12.254550 8602 ts_tablet_manager.cc:1403] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:12.254850 8603 tablet_bootstrap.cc:492] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4: No bootstrap required, opened a new log
I20260131 03:02:12.254951 8603 ts_tablet_manager.cc:1403] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:12.254971 8602 raft_consensus.cc:359] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.255070 8602 raft_consensus.cc:385] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:12.255101 8602 raft_consensus.cc:740] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519e1d3f6c1645a3964028dd4efd4129, State: Initialized, Role: FOLLOWER
I20260131 03:02:12.255281 8602 consensus_queue.cc:260] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.255393 8604 tablet_bootstrap.cc:654] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:12.255406 8603 raft_consensus.cc:359] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.255523 8603 raft_consensus.cc:385] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:12.255561 8603 raft_consensus.cc:740] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b8b0afc91f4df189029455f588a7d4, State: Initialized, Role: FOLLOWER
I20260131 03:02:12.255681 8603 consensus_queue.cc:260] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.256043 8603 ts_tablet_manager.cc:1434] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:12.256771 8604 tablet_bootstrap.cc:492] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602: No bootstrap required, opened a new log
I20260131 03:02:12.256865 8604 ts_tablet_manager.cc:1403] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:12.257284 8602 ts_tablet_manager.cc:1434] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260131 03:02:12.257277 8604 raft_consensus.cc:359] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.257558 8604 raft_consensus.cc:385] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:12.257614 8604 raft_consensus.cc:740] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894437dcf1ec4f40826976f686fab602, State: Initialized, Role: FOLLOWER
I20260131 03:02:12.257742 8604 consensus_queue.cc:260] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.258033 8604 ts_tablet_manager.cc:1434] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:12.297469 8528 raft_consensus.cc:493] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:02:12.297601 8528 raft_consensus.cc:515] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.297953 8528 leader_election.cc:290] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261), f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513)
I20260131 03:02:12.298398 8153 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d691317f7f345fa8b9a7a271d980eba" candidate_uuid: "519e1d3f6c1645a3964028dd4efd4129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894437dcf1ec4f40826976f686fab602" is_pre_election: true
I20260131 03:02:12.298398 8279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d691317f7f345fa8b9a7a271d980eba" candidate_uuid: "519e1d3f6c1645a3964028dd4efd4129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b8b0afc91f4df189029455f588a7d4" is_pre_election: true
I20260131 03:02:12.298548 8279 raft_consensus.cc:2468] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 519e1d3f6c1645a3964028dd4efd4129 in term 0.
I20260131 03:02:12.298547 8153 raft_consensus.cc:2468] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 519e1d3f6c1645a3964028dd4efd4129 in term 0.
I20260131 03:02:12.298933 8371 leader_election.cc:304] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519e1d3f6c1645a3964028dd4efd4129, f8b8b0afc91f4df189029455f588a7d4; no voters:
I20260131 03:02:12.299180 8528 raft_consensus.cc:2804] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:02:12.299274 8528 raft_consensus.cc:493] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:02:12.299331 8528 raft_consensus.cc:3060] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:12.300298 8528 raft_consensus.cc:515] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.300635 8528 leader_election.cc:290] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [CANDIDATE]: Term 1 election: Requested vote from peers 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261), f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513)
I20260131 03:02:12.300999 8153 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d691317f7f345fa8b9a7a271d980eba" candidate_uuid: "519e1d3f6c1645a3964028dd4efd4129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894437dcf1ec4f40826976f686fab602"
I20260131 03:02:12.301018 8279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d691317f7f345fa8b9a7a271d980eba" candidate_uuid: "519e1d3f6c1645a3964028dd4efd4129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b8b0afc91f4df189029455f588a7d4"
I20260131 03:02:12.301129 8153 raft_consensus.cc:3060] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:12.301151 8279 raft_consensus.cc:3060] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:12.302191 8153 raft_consensus.cc:2468] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 519e1d3f6c1645a3964028dd4efd4129 in term 1.
I20260131 03:02:12.302227 8279 raft_consensus.cc:2468] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 519e1d3f6c1645a3964028dd4efd4129 in term 1.
I20260131 03:02:12.302520 8368 leader_election.cc:304] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519e1d3f6c1645a3964028dd4efd4129, 894437dcf1ec4f40826976f686fab602; no voters:
I20260131 03:02:12.302704 8528 raft_consensus.cc:2804] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:12.302804 8528 raft_consensus.cc:697] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 1 LEADER]: Becoming Leader. State: Replica: 519e1d3f6c1645a3964028dd4efd4129, State: Running, Role: LEADER
I20260131 03:02:12.302950 8528 consensus_queue.cc:237] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.304472 7991 catalog_manager.cc:5654] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 reported cstate change: term changed from 0 to 1, leader changed from <none> to 519e1d3f6c1645a3964028dd4efd4129 (127.6.22.67). New cstate: current_term: 1 leader_uuid: "519e1d3f6c1645a3964028dd4efd4129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } health_report { overall_health: HEALTHY } } }
I20260131 03:02:12.326382 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:12.323463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.6.22.67:40073)
0131 03:02:12.323718 (+ 255us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:12.323722 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:12.323734 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:02:12.323770 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:12.323773 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:12.323816 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:12.323887 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:12.324210 (+ 323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:12.325024 (+ 814us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:12.325721 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:12.325906 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:12.325950 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:12.326025 (+ 75us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:12.326099 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:12.326199 (+ 100us) server_negotiation.cc:300] Negotiation successful
0131 03:02:12.326247 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":118,"threads_started":1}
I20260131 03:02:12.336601 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:12.333073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:41299 (local address 127.6.22.67:40073)
0131 03:02:12.333255 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:12.333260 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:12.333277 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:02:12.333529 (+ 252us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:12.333540 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:12.333612 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:12.333715 (+ 103us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:12.334088 (+ 373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:12.334838 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:12.335820 (+ 982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:12.336013 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:12.336059 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:12.336130 (+ 71us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:12.336214 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:12.336383 (+ 169us) server_negotiation.cc:300] Negotiation successful
0131 03:02:12.336432 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260131 03:02:12.336608 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:12.332925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:40073 (local address 127.6.22.67:41299)
0131 03:02:12.333208 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:12.333223 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:12.333305 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:12.333752 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:12.333755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:12.333763 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:02:12.333965 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:12.333970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:12.335195 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:12.335199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:12.335685 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:12.335693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:12.335810 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:12.335840 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:02:12.336273 (+ 433us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:02:12.336279 (+ 6us) client_negotiation.cc:769] Sending connection context
0131 03:02:12.336361 (+ 82us) client_negotiation.cc:240] Negotiation successful
0131 03:02:12.336439 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":113,"threads_started":1}
I20260131 03:02:12.342280 8279 raft_consensus.cc:1275] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Refusing update from remote peer 519e1d3f6c1645a3964028dd4efd4129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:12.342648 8153 raft_consensus.cc:1275] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Refusing update from remote peer 519e1d3f6c1645a3964028dd4efd4129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:12.342666 8528 consensus_queue.cc:1048] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:12.343029 8528 consensus_queue.cc:1048] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:12.375082 7994 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:12.375249 7994 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:12.377203 7994 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480: Sending DeleteTablet for 3 replicas of tablet 8d691317f7f345fa8b9a7a271d980eba
I20260131 03:02:12.377774 8417 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d691317f7f345fa8b9a7a271d980eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:12 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:51224
I20260131 03:02:12.377794 8275 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d691317f7f345fa8b9a7a271d980eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:12 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:56944
I20260131 03:02:12.377789 8133 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d691317f7f345fa8b9a7a271d980eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:12 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:49222
I20260131 03:02:12.377928 8593 tablet_replica.cc:333] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129: stopping tablet replica
I20260131 03:02:12.377948 8591 tablet_replica.cc:333] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602: stopping tablet replica
I20260131 03:02:12.377948 8592 tablet_replica.cc:333] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4: stopping tablet replica
I20260131 03:02:12.378026 8593 raft_consensus.cc:2243] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:02:12.378043 8592 raft_consensus.cc:2243] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:12.378043 8591 raft_consensus.cc:2243] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:12.378137 8591 raft_consensus.cc:2272] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:12.378137 8592 raft_consensus.cc:2272] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:12.378258 8593 raft_consensus.cc:2272] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:12.378908 8593 ts_tablet_manager.cc:1916] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:12.379189 8591 ts_tablet_manager.cc:1916] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:12.381021 8591 ts_tablet_manager.cc:1929] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:02:12.381114 8591 log.cc:1199] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-0/wal/wals/8d691317f7f345fa8b9a7a271d980eba
I20260131 03:02:12.381232 8593 ts_tablet_manager.cc:1929] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:02:12.381307 8593 log.cc:1199] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/8d691317f7f345fa8b9a7a271d980eba
I20260131 03:02:12.381393 8591 ts_tablet_manager.cc:1950] T 8d691317f7f345fa8b9a7a271d980eba P 894437dcf1ec4f40826976f686fab602: Deleting consensus metadata
I20260131 03:02:12.381598 8593 ts_tablet_manager.cc:1950] T 8d691317f7f345fa8b9a7a271d980eba P 519e1d3f6c1645a3964028dd4efd4129: Deleting consensus metadata
I20260131 03:02:12.382081 7978 catalog_manager.cc:4985] TS 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261): tablet 8d691317f7f345fa8b9a7a271d980eba (table test-table [id=b84096c0e8dd47e18e587320add3d4ba]) successfully deleted
I20260131 03:02:12.382225 7978 catalog_manager.cc:4985] TS 519e1d3f6c1645a3964028dd4efd4129 (127.6.22.67:40073): tablet 8d691317f7f345fa8b9a7a271d980eba (table test-table [id=b84096c0e8dd47e18e587320add3d4ba]) successfully deleted
I20260131 03:02:12.382258 7994 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260131 03:02:12.382586 8592 ts_tablet_manager.cc:1916] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260131 03:02:12.382624 7994 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:02:12.384832 8592 ts_tablet_manager.cc:1929] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:02:12.384902 8592 log.cc:1199] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/8d691317f7f345fa8b9a7a271d980eba
I20260131 03:02:12.385150 8592 ts_tablet_manager.cc:1950] T 8d691317f7f345fa8b9a7a271d980eba P f8b8b0afc91f4df189029455f588a7d4: Deleting consensus metadata
I20260131 03:02:12.385717 7981 catalog_manager.cc:4985] TS f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513): tablet 8d691317f7f345fa8b9a7a271d980eba (table test-table [id=b84096c0e8dd47e18e587320add3d4ba]) successfully deleted
I20260131 03:02:12.386969 8133 tablet_service.cc:1505] Processing CreateTablet for tablet 8e4e1789eb3c4da9b47fb3cc00d146d9 (DEFAULT_TABLE table=test-table [id=97ffd7781fc740b1a4c6dc58f9db0f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:12.387187 8133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e4e1789eb3c4da9b47fb3cc00d146d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:12.387234 8275 tablet_service.cc:1505] Processing CreateTablet for tablet 8e4e1789eb3c4da9b47fb3cc00d146d9 (DEFAULT_TABLE table=test-table [id=97ffd7781fc740b1a4c6dc58f9db0f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:12.387455 8275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e4e1789eb3c4da9b47fb3cc00d146d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:12.388886 8604 tablet_bootstrap.cc:492] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602: Bootstrap starting.
I20260131 03:02:12.389438 8603 tablet_bootstrap.cc:492] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4: Bootstrap starting.
I20260131 03:02:12.389642 8417 tablet_service.cc:1505] Processing CreateTablet for tablet 8e4e1789eb3c4da9b47fb3cc00d146d9 (DEFAULT_TABLE table=test-table [id=97ffd7781fc740b1a4c6dc58f9db0f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:12.389771 8604 tablet_bootstrap.cc:654] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:12.389850 8417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e4e1789eb3c4da9b47fb3cc00d146d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:12.390317 8603 tablet_bootstrap.cc:654] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:12.390805 8604 tablet_bootstrap.cc:492] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602: No bootstrap required, opened a new log
I20260131 03:02:12.390897 8604 ts_tablet_manager.cc:1403] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:02:12.391351 8604 raft_consensus.cc:359] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.391458 8603 tablet_bootstrap.cc:492] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4: No bootstrap required, opened a new log
I20260131 03:02:12.391450 8604 raft_consensus.cc:385] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:12.391489 8604 raft_consensus.cc:740] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894437dcf1ec4f40826976f686fab602, State: Initialized, Role: FOLLOWER
I20260131 03:02:12.391525 8603 ts_tablet_manager.cc:1403] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:12.391618 8604 consensus_queue.cc:260] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.391858 8604 ts_tablet_manager.cc:1434] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:12.391894 8603 raft_consensus.cc:359] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.391976 8603 raft_consensus.cc:385] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:12.392009 8603 raft_consensus.cc:740] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b8b0afc91f4df189029455f588a7d4, State: Initialized, Role: FOLLOWER
I20260131 03:02:12.392127 8603 consensus_queue.cc:260] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.392277 8602 tablet_bootstrap.cc:492] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129: Bootstrap starting.
I20260131 03:02:12.392392 8603 ts_tablet_manager.cc:1434] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:12.393108 8602 tablet_bootstrap.cc:654] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:12.394093 8602 tablet_bootstrap.cc:492] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129: No bootstrap required, opened a new log
I20260131 03:02:12.394196 8602 ts_tablet_manager.cc:1403] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:02:12.394632 8602 raft_consensus.cc:359] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.394731 8602 raft_consensus.cc:385] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:12.394770 8602 raft_consensus.cc:740] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519e1d3f6c1645a3964028dd4efd4129, State: Initialized, Role: FOLLOWER
I20260131 03:02:12.394901 8602 consensus_queue.cc:260] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.395184 8602 ts_tablet_manager.cc:1434] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260131 03:02:12.456396 8577 raft_consensus.cc:493] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:02:12.456540 8577 raft_consensus.cc:515] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.457017 8577 leader_election.cc:290] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261), f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513)
I20260131 03:02:12.457420 8279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e4e1789eb3c4da9b47fb3cc00d146d9" candidate_uuid: "519e1d3f6c1645a3964028dd4efd4129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b8b0afc91f4df189029455f588a7d4" is_pre_election: true
I20260131 03:02:12.457444 8152 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e4e1789eb3c4da9b47fb3cc00d146d9" candidate_uuid: "519e1d3f6c1645a3964028dd4efd4129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894437dcf1ec4f40826976f686fab602" is_pre_election: true
I20260131 03:02:12.457580 8152 raft_consensus.cc:2468] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 519e1d3f6c1645a3964028dd4efd4129 in term 0.
I20260131 03:02:12.457589 8279 raft_consensus.cc:2468] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 519e1d3f6c1645a3964028dd4efd4129 in term 0.
I20260131 03:02:12.457924 8368 leader_election.cc:304] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519e1d3f6c1645a3964028dd4efd4129, 894437dcf1ec4f40826976f686fab602; no voters:
I20260131 03:02:12.458078 8577 raft_consensus.cc:2804] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:02:12.458146 8577 raft_consensus.cc:493] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:02:12.458202 8577 raft_consensus.cc:3060] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:12.459067 8577 raft_consensus.cc:515] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.459441 8577 leader_election.cc:290] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [CANDIDATE]: Term 1 election: Requested vote from peers 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261), f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513)
I20260131 03:02:12.459734 8152 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e4e1789eb3c4da9b47fb3cc00d146d9" candidate_uuid: "519e1d3f6c1645a3964028dd4efd4129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894437dcf1ec4f40826976f686fab602"
I20260131 03:02:12.459789 8279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e4e1789eb3c4da9b47fb3cc00d146d9" candidate_uuid: "519e1d3f6c1645a3964028dd4efd4129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b8b0afc91f4df189029455f588a7d4"
I20260131 03:02:12.459843 8152 raft_consensus.cc:3060] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:12.459926 8279 raft_consensus.cc:3060] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:12.460819 8279 raft_consensus.cc:2468] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 519e1d3f6c1645a3964028dd4efd4129 in term 1.
I20260131 03:02:12.460928 8152 raft_consensus.cc:2468] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 519e1d3f6c1645a3964028dd4efd4129 in term 1.
I20260131 03:02:12.461138 8371 leader_election.cc:304] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519e1d3f6c1645a3964028dd4efd4129, f8b8b0afc91f4df189029455f588a7d4; no voters:
I20260131 03:02:12.461366 8577 raft_consensus.cc:2804] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:12.461460 8577 raft_consensus.cc:697] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 1 LEADER]: Becoming Leader. State: Replica: 519e1d3f6c1645a3964028dd4efd4129, State: Running, Role: LEADER
I20260131 03:02:12.461606 8577 consensus_queue.cc:237] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } }
I20260131 03:02:12.463363 7991 catalog_manager.cc:5654] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 reported cstate change: term changed from 0 to 1, leader changed from <none> to 519e1d3f6c1645a3964028dd4efd4129 (127.6.22.67). New cstate: current_term: 1 leader_uuid: "519e1d3f6c1645a3964028dd4efd4129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "519e1d3f6c1645a3964028dd4efd4129" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 40073 } health_report { overall_health: HEALTHY } } }
I20260131 03:02:12.481966 8153 raft_consensus.cc:1275] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Refusing update from remote peer 519e1d3f6c1645a3964028dd4efd4129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:12.482131 8295 raft_consensus.cc:1275] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Refusing update from remote peer 519e1d3f6c1645a3964028dd4efd4129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:12.482453 8528 consensus_queue.cc:1048] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894437dcf1ec4f40826976f686fab602" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:12.482826 8528 consensus_queue.cc:1048] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8b8b0afc91f4df189029455f588a7d4" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260131 03:02:12.496588 8570 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260131 03:02:14.045711 7991 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:14.045855 7991 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:14.048027 7991 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 4f68701b7f7d40dfbcd97e4239095480: Sending DeleteTablet for 3 replicas of tablet 8e4e1789eb3c4da9b47fb3cc00d146d9
I20260131 03:02:14.048594 8133 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e4e1789eb3c4da9b47fb3cc00d146d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:14 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:49222
I20260131 03:02:14.048710 8275 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e4e1789eb3c4da9b47fb3cc00d146d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:14 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:56944
I20260131 03:02:14.048784 8416 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e4e1789eb3c4da9b47fb3cc00d146d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:14 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:51224
I20260131 03:02:14.048975 8644 tablet_replica.cc:333] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602: stopping tablet replica
I20260131 03:02:14.048995 8645 tablet_replica.cc:333] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4: stopping tablet replica
I20260131 03:02:14.049118 8645 raft_consensus.cc:2243] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:14.049139 8644 raft_consensus.cc:2243] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:14.049185 8646 tablet_replica.cc:333] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129: stopping tablet replica
I20260131 03:02:14.049256 8644 raft_consensus.cc:2272] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 894437dcf1ec4f40826976f686fab602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:14.049292 8646 raft_consensus.cc:2243] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:02:14.049394 8645 raft_consensus.cc:2272] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:14.049527 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 8068
I20260131 03:02:14.050386 8646 raft_consensus.cc:2272] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:14.051107 8646 ts_tablet_manager.cc:1916] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:14.051720 8645 ts_tablet_manager.cc:1916] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:14.053720 8646 ts_tablet_manager.cc:1929] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260131 03:02:14.053799 8646 log.cc:1199] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/8e4e1789eb3c4da9b47fb3cc00d146d9
I20260131 03:02:14.054083 8646 ts_tablet_manager.cc:1950] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P 519e1d3f6c1645a3964028dd4efd4129: Deleting consensus metadata
I20260131 03:02:14.054350 8645 ts_tablet_manager.cc:1929] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260131 03:02:14.054425 8645 log.cc:1199] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/8e4e1789eb3c4da9b47fb3cc00d146d9
I20260131 03:02:14.054755 8645 ts_tablet_manager.cc:1950] T 8e4e1789eb3c4da9b47fb3cc00d146d9 P f8b8b0afc91f4df189029455f588a7d4: Deleting consensus metadata
I20260131 03:02:14.054795 7978 catalog_manager.cc:4985] TS 519e1d3f6c1645a3964028dd4efd4129 (127.6.22.67:40073): tablet 8e4e1789eb3c4da9b47fb3cc00d146d9 (table test-table [id=97ffd7781fc740b1a4c6dc58f9db0f28]) successfully deleted
I20260131 03:02:14.055394 7981 catalog_manager.cc:4985] TS f8b8b0afc91f4df189029455f588a7d4 (127.6.22.66:41513): tablet 8e4e1789eb3c4da9b47fb3cc00d146d9 (table test-table [id=97ffd7781fc740b1a4c6dc58f9db0f28]) successfully deleted
W20260131 03:02:14.057608 7978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.22.65:35261 (error 108)
I20260131 03:02:14.057782 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 8210
I20260131 03:02:14.059289 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:14.058813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:35261 (local address 127.0.0.1:49232)
0131 03:02:14.059085 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:14.059160 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.65:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
W20260131 03:02:14.059589 7978 catalog_manager.cc:4712] TS 894437dcf1ec4f40826976f686fab602 (127.6.22.65:35261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e4e1789eb3c4da9b47fb3cc00d146d9: Network error: Client connection negotiation failed: client connection to 127.6.22.65:35261: connect: Connection refused (error 111)
I20260131 03:02:14.067305 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 8352
I20260131 03:02:14.075150 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 7962
2026-01-31T03:02:14Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7161 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:02:14 dist-test-slave-j368 krb5kdc[8651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:02:14 dist-test-slave-j368 krb5kdc[8651](info): set up 2 sockets
Jan 31 03:02:14 dist-test-slave-j368 krb5kdc[8651](info): commencing operation
krb5kdc: starting...
W20260131 03:02:16.038000 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:02:16 dist-test-slave-j368 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:02:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:16.172423 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:42321
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:42321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:16.286995 8667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:16.287281 8667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:16.287379 8667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:16.290635 8667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:16.290728 8667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:16.290781 8667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:16.290839 8667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:16.290882 8667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:16.295051 8667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:42321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:42321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:16.296252 8667 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:16.297181 8667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:16.303404 8673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:16.303344 8675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:16.303558 8667 server_base.cc:1060] running on GCE node
W20260131 03:02:16.303344 8672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:16.304236 8667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:16.305245 8667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:16.306468 8667 hybrid_clock.cc:648] HybridClock initialized: now 1769828536306387 us; error 98 us; skew 500 ppm
Jan 31 03:02:16 dist-test-slave-j368 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:16.309669 8667 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:02:16.310992 8667 webserver.cc:492] Webserver started at http://127.6.22.126:38915/ using document root <none> and password file <none>
I20260131 03:02:16.311587 8667 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:16.311642 8667 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:16.311868 8667 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:16.313741 8667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "69e33dad0d4b44a59ab9e019d69d1a3a"
format_stamp: "Formatted at 2026-01-31 03:02:16 on dist-test-slave-j368"
server_key: "7bd3c74947e4dd8512205d03635ba8e7"
server_key_iv: "9a230194f5237ea3cc01209c26d60851"
server_key_version: "encryptionkey@0"
I20260131 03:02:16.314247 8667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "69e33dad0d4b44a59ab9e019d69d1a3a"
format_stamp: "Formatted at 2026-01-31 03:02:16 on dist-test-slave-j368"
server_key: "7bd3c74947e4dd8512205d03635ba8e7"
server_key_iv: "9a230194f5237ea3cc01209c26d60851"
server_key_version: "encryptionkey@0"
I20260131 03:02:16.318290 8667 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260131 03:02:16.320979 8682 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:16.322041 8667 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260131 03:02:16.322175 8667 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "69e33dad0d4b44a59ab9e019d69d1a3a"
format_stamp: "Formatted at 2026-01-31 03:02:16 on dist-test-slave-j368"
server_key: "7bd3c74947e4dd8512205d03635ba8e7"
server_key_iv: "9a230194f5237ea3cc01209c26d60851"
server_key_version: "encryptionkey@0"
I20260131 03:02:16.322285 8667 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:16.352083 8667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:16.355129 8667 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:16.355356 8667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:16.363086 8667 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:42321
I20260131 03:02:16.363095 8734 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:42321 every 8 connection(s)
I20260131 03:02:16.364097 8667 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:16.367987 8735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:16.369488 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 8667
I20260131 03:02:16.369650 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:16.369978 6233 external_mini_cluster.cc:1468] Setting key 51f9ed636dcef7af380a7729497182cd
I20260131 03:02:16.376080 8735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a: Bootstrap starting.
Jan 31 03:02:16 dist-test-slave-j368 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:16.379451 8735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:16.380211 8735 log.cc:826] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:16.382464 8735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a: No bootstrap required, opened a new log
I20260131 03:02:16.384999 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:16.371604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.6.22.126:42321)
0131 03:02:16.372126 (+ 522us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:16.372136 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:16.372178 (+ 42us) server_negotiation.cc:408] Connection header received
0131 03:02:16.372944 (+ 766us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:16.372965 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:16.373303 (+ 338us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:16.373636 (+ 333us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:16.374828 (+ 1192us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:16.375637 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:16.376543 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:16.376867 (+ 324us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:16.379777 (+ 2910us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:16.379794 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:16.379804 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:16.379842 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:16.381958 (+ 2116us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:16.382418 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:16.382424 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:16.382429 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:16.382530 (+ 101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:16.382872 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:16.382875 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:16.382877 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:16.383319 (+ 442us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:16.383530 (+ 211us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:16.383893 (+ 363us) server_negotiation.cc:300] Negotiation successful
0131 03:02:16.384169 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":173,"threads_started":1}
I20260131 03:02:16.386049 8735 raft_consensus.cc:359] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e33dad0d4b44a59ab9e019d69d1a3a" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 42321 } }
I20260131 03:02:16.386276 8735 raft_consensus.cc:385] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:16.386323 8735 raft_consensus.cc:740] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e33dad0d4b44a59ab9e019d69d1a3a, State: Initialized, Role: FOLLOWER
I20260131 03:02:16.386797 8735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e33dad0d4b44a59ab9e019d69d1a3a" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 42321 } }
I20260131 03:02:16.386914 8735 raft_consensus.cc:399] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:16.386992 8735 raft_consensus.cc:493] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:16.387115 8735 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:16.388353 8735 raft_consensus.cc:515] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e33dad0d4b44a59ab9e019d69d1a3a" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 42321 } }
I20260131 03:02:16.388731 8735 leader_election.cc:304] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69e33dad0d4b44a59ab9e019d69d1a3a; no voters:
I20260131 03:02:16.389038 8735 leader_election.cc:290] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:16.389182 8740 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:16.389389 8740 raft_consensus.cc:697] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [term 1 LEADER]: Becoming Leader. State: Replica: 69e33dad0d4b44a59ab9e019d69d1a3a, State: Running, Role: LEADER
I20260131 03:02:16.389712 8740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e33dad0d4b44a59ab9e019d69d1a3a" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 42321 } }
I20260131 03:02:16.390209 8735 sys_catalog.cc:565] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:16.391269 8742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e33dad0d4b44a59ab9e019d69d1a3a. Latest consensus state: current_term: 1 leader_uuid: "69e33dad0d4b44a59ab9e019d69d1a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e33dad0d4b44a59ab9e019d69d1a3a" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 42321 } } }
I20260131 03:02:16.391391 8742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:16.391902 8741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69e33dad0d4b44a59ab9e019d69d1a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e33dad0d4b44a59ab9e019d69d1a3a" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 42321 } } }
I20260131 03:02:16.391999 8741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a [sys.catalog]: This master's current role is: LEADER
W20260131 03:02:16.394742 8755 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:02:16.394815 8755 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:02:16.394903 8749 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:16.395676 8749 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:16.400785 8749 catalog_manager.cc:1357] Generated new cluster ID: 3f28cc1f9b874ea889e8545940cb7d17
I20260131 03:02:16.400863 8749 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:16.406281 8749 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:16.407156 8749 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:16.418298 8749 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 69e33dad0d4b44a59ab9e019d69d1a3a: Generated new TSK 0
I20260131 03:02:16.419049 8749 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:16.480571 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:42321
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:16.594111 8763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:16.594362 8763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:16.594424 8763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:16.597855 8763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:16.597934 8763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:16.598021 8763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:02:16.602500 8763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:42321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:16.603636 8763 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:16.604475 8763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:16.611800 8768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:16.611867 8769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:16.611845 8771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:16.612030 8763 server_base.cc:1060] running on GCE node
I20260131 03:02:16.612385 8763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:16.613041 8763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:16.614204 8763 hybrid_clock.cc:648] HybridClock initialized: now 1769828536614180 us; error 38 us; skew 500 ppm
Jan 31 03:02:16 dist-test-slave-j368 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:16.617128 8763 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:02:16.618225 8763 webserver.cc:492] Webserver started at http://127.6.22.65:46385/ using document root <none> and password file <none>
I20260131 03:02:16.618824 8763 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:16.618901 8763 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:16.619117 8763 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:16.620970 8763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "270ab21f40234b779d23c8f9ef0eeedd"
format_stamp: "Formatted at 2026-01-31 03:02:16 on dist-test-slave-j368"
server_key: "8e4ade8f267c1856324be9d53114b3b3"
server_key_iv: "0bcf91c1d0cef8c118d4405ded2972e0"
server_key_version: "encryptionkey@0"
I20260131 03:02:16.621486 8763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "270ab21f40234b779d23c8f9ef0eeedd"
format_stamp: "Formatted at 2026-01-31 03:02:16 on dist-test-slave-j368"
server_key: "8e4ade8f267c1856324be9d53114b3b3"
server_key_iv: "0bcf91c1d0cef8c118d4405ded2972e0"
server_key_version: "encryptionkey@0"
I20260131 03:02:16.625082 8763 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:02:16.627671 8778 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:16.629352 8763 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:02:16.629580 8763 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "270ab21f40234b779d23c8f9ef0eeedd"
format_stamp: "Formatted at 2026-01-31 03:02:16 on dist-test-slave-j368"
server_key: "8e4ade8f267c1856324be9d53114b3b3"
server_key_iv: "0bcf91c1d0cef8c118d4405ded2972e0"
server_key_version: "encryptionkey@0"
I20260131 03:02:16.629737 8763 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:16.666458 8763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:16.670295 8763 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:16.670524 8763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:16.671108 8763 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:16.672143 8763 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:16.672214 8763 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:16.672289 8763 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:16.672343 8763 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:16.682284 8763 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:46685
I20260131 03:02:16.682547 8891 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:46685 every 8 connection(s)
I20260131 03:02:16.683516 8763 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:02:16.687806 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 8763
I20260131 03:02:16.687909 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:02:16.688232 6233 external_mini_cluster.cc:1468] Setting key a460f4a50c56327c1861c3ff1b3e9999
Jan 31 03:02:16 dist-test-slave-j368 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
I20260131 03:02:16.698377 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:16.685890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:55203 (local address 127.6.22.126:42321)
0131 03:02:16.686077 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:16.686081 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:16.686916 (+ 835us) server_negotiation.cc:408] Connection header received
0131 03:02:16.688062 (+ 1146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:16.688067 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:16.688141 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:16.688252 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:16.690429 (+ 2177us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:16.690951 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:16.691757 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:16.691959 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:16.694508 (+ 2549us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:16.694531 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:16.694534 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:16.694562 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:16.695997 (+ 1435us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:16.696764 (+ 767us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:16.696767 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:16.696769 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:16.696817 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:16.697172 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:16.697174 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:16.697175 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:16.697333 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:16.697408 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:16.698118 (+ 710us) server_negotiation.cc:300] Negotiation successful
0131 03:02:16.698246 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260131 03:02:16.699185 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:16.686144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:42321 (local address 127.6.22.65:55203)
0131 03:02:16.686693 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:16.686741 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:16.687751 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:16.688610 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:16.688627 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:16.689145 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:16.690249 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:16.690269 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:16.691093 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:16.691097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:16.691630 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:16.691639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:16.691887 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:16.692605 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:16.692631 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:16.694319 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:16.696187 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:16.696192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:16.696207 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:16.696577 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:16.696923 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:16.696926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:16.696929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:16.697064 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:16.697439 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:16.697454 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:16.697806 (+ 352us) client_negotiation.cc:769] Sending connection context
0131 03:02:16.698073 (+ 267us) client_negotiation.cc:240] Negotiation successful
0131 03:02:16.698321 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
I20260131 03:02:16.700484 8892 heartbeater.cc:344] Connected to a master server at 127.6.22.126:42321
I20260131 03:02:16.700806 8892 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:16.701357 8892 heartbeater.cc:507] Master 127.6.22.126:42321 requested a full tablet report, sending...
I20260131 03:02:16.703018 8699 ts_manager.cc:194] Registered new tserver with Master: 270ab21f40234b779d23c8f9ef0eeedd (127.6.22.65:46685)
I20260131 03:02:16.704746 8699 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:55203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:16.743782 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:42321
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:16.854210 8899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:16.854454 8899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:16.854516 8899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:16.857836 8899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:16.857905 8899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:16.857986 8899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:02:16.862388 8899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:42321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:16.863550 8899 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:16.864384 8899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:16.871037 8904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:16.870973 8907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:16.871214 8899 server_base.cc:1060] running on GCE node
W20260131 03:02:16.870973 8905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:16.871686 8899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:16.872255 8899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:16.873469 8899 hybrid_clock.cc:648] HybridClock initialized: now 1769828536873442 us; error 42 us; skew 500 ppm
Jan 31 03:02:16 dist-test-slave-j368 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:16.876503 8899 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:02:16.877661 8899 webserver.cc:492] Webserver started at http://127.6.22.66:33079/ using document root <none> and password file <none>
I20260131 03:02:16.878245 8899 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:16.878324 8899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:16.878535 8899 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:16.880359 8899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "d99009ad0be84ad5b6bf6c412f6f891c"
format_stamp: "Formatted at 2026-01-31 03:02:16 on dist-test-slave-j368"
server_key: "928055f55537806352e6cad4ff38de22"
server_key_iv: "dbbc18e4f74854aab18177cbf59b0c08"
server_key_version: "encryptionkey@0"
I20260131 03:02:16.880913 8899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "d99009ad0be84ad5b6bf6c412f6f891c"
format_stamp: "Formatted at 2026-01-31 03:02:16 on dist-test-slave-j368"
server_key: "928055f55537806352e6cad4ff38de22"
server_key_iv: "dbbc18e4f74854aab18177cbf59b0c08"
server_key_version: "encryptionkey@0"
I20260131 03:02:16.884984 8899 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260131 03:02:16.887362 8914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:16.888585 8899 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:16.888759 8899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "d99009ad0be84ad5b6bf6c412f6f891c"
format_stamp: "Formatted at 2026-01-31 03:02:16 on dist-test-slave-j368"
server_key: "928055f55537806352e6cad4ff38de22"
server_key_iv: "dbbc18e4f74854aab18177cbf59b0c08"
server_key_version: "encryptionkey@0"
I20260131 03:02:16.888888 8899 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:16.903956 8899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:16.907898 8899 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:16.908144 8899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:16.908808 8899 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:16.909701 8899 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:16.909780 8899 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:16.909863 8899 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:16.909911 8899 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:16.920217 8899 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:43277
I20260131 03:02:16.921375 8899 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:02:16.922984 9027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:43277 every 8 connection(s)
I20260131 03:02:16.931083 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 8899
I20260131 03:02:16.931218 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:02:16.931486 6233 external_mini_cluster.cc:1468] Setting key b8aa7fdf7f1daa4978cce0fed512f408
Jan 31 03:02:16 dist-test-slave-j368 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:16.936975 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:16.923532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:56653 (local address 127.6.22.126:42321)
0131 03:02:16.923737 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:16.923744 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:16.924470 (+ 726us) server_negotiation.cc:408] Connection header received
0131 03:02:16.925687 (+ 1217us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:16.925693 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:16.925754 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:16.925854 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:16.928287 (+ 2433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:16.929103 (+ 816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:16.929941 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:16.930124 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:16.932938 (+ 2814us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:16.932956 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:16.932962 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:16.933001 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:16.934642 (+ 1641us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:16.935149 (+ 507us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:16.935155 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:16.935159 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:16.935213 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:16.935667 (+ 454us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:16.935673 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:16.935678 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:16.935912 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:16.936017 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:16.936596 (+ 579us) server_negotiation.cc:300] Negotiation successful
0131 03:02:16.936779 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260131 03:02:16.937654 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:16.923795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:42321 (local address 127.6.22.66:56653)
0131 03:02:16.924315 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:16.924365 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:16.925449 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:16.926047 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:16.926061 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:16.926704 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:16.928087 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:16.928107 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:16.929329 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:16.929332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:16.929754 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:16.929761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:16.929978 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:16.930638 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:16.930664 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:16.932747 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:16.934782 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:16.934787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:16.934797 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:16.935046 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:16.935422 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:16.935426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:16.935428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:16.935547 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:16.936036 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:16.936046 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:16.936308 (+ 262us) client_negotiation.cc:769] Sending connection context
0131 03:02:16.936521 (+ 213us) client_negotiation.cc:240] Negotiation successful
0131 03:02:16.936840 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":95,"threads_started":1}
I20260131 03:02:16.938819 9028 heartbeater.cc:344] Connected to a master server at 127.6.22.126:42321
I20260131 03:02:16.939108 9028 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:16.939801 9028 heartbeater.cc:507] Master 127.6.22.126:42321 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
I20260131 03:02:16.941098 8699 ts_manager.cc:194] Registered new tserver with Master: d99009ad0be84ad5b6bf6c412f6f891c (127.6.22.66:43277)
I20260131 03:02:16.941697 8699 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:56653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:16.982015 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:42321
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:17.099812 9035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:17.100018 9035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:17.100104 9035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:17.103457 9035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:17.103511 9035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:17.103632 9035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:02:17.107780 9035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:42321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:17.108928 9035 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:17.109709 9035 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:17.115964 9043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:17.115958 9041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:17.115958 9040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:17.116322 9035 server_base.cc:1060] running on GCE node
I20260131 03:02:17.116742 9035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:17.117261 9035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:17.118484 9035 hybrid_clock.cc:648] HybridClock initialized: now 1769828537118475 us; error 45 us; skew 500 ppm
Jan 31 03:02:17 dist-test-slave-j368 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:17.121004 9035 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:02:17.121933 9035 webserver.cc:492] Webserver started at http://127.6.22.67:33051/ using document root <none> and password file <none>
I20260131 03:02:17.122426 9035 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:17.122471 9035 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:17.122612 9035 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:17.124194 9035 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "2b6513ffe306430c8f3bd201a31e3837"
format_stamp: "Formatted at 2026-01-31 03:02:17 on dist-test-slave-j368"
server_key: "8e045c29bce23832c88d8dc6b277540d"
server_key_iv: "7c409e87873eb0abac177d61cc6ccbb3"
server_key_version: "encryptionkey@0"
I20260131 03:02:17.124611 9035 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "2b6513ffe306430c8f3bd201a31e3837"
format_stamp: "Formatted at 2026-01-31 03:02:17 on dist-test-slave-j368"
server_key: "8e045c29bce23832c88d8dc6b277540d"
server_key_iv: "7c409e87873eb0abac177d61cc6ccbb3"
server_key_version: "encryptionkey@0"
I20260131 03:02:17.127954 9035 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260131 03:02:17.130028 9050 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:17.130983 9035 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:02:17.131069 9035 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "2b6513ffe306430c8f3bd201a31e3837"
format_stamp: "Formatted at 2026-01-31 03:02:17 on dist-test-slave-j368"
server_key: "8e045c29bce23832c88d8dc6b277540d"
server_key_iv: "7c409e87873eb0abac177d61cc6ccbb3"
server_key_version: "encryptionkey@0"
I20260131 03:02:17.131139 9035 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:17.144011 9035 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:17.146428 9035 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:17.146567 9035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:17.147029 9035 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:17.147826 9035 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:17.147871 9035 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:17.147909 9035 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:17.147929 9035 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:17.157019 9035 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:36503
I20260131 03:02:17.157040 9163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:36503 every 8 connection(s)
I20260131 03:02:17.157896 9035 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:02:17.158139 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 9035
I20260131 03:02:17.158252 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:02:17.158514 6233 external_mini_cluster.cc:1468] Setting key a42e760396c81218e2a7a7ec985d7e27
Jan 31 03:02:17 dist-test-slave-j368 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:17.169569 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:17.159658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:34329 (local address 127.6.22.126:42321)
0131 03:02:17.159793 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:17.159797 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:17.160512 (+ 715us) server_negotiation.cc:408] Connection header received
0131 03:02:17.161423 (+ 911us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:17.161425 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:17.161471 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:17.161540 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:17.163135 (+ 1595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:17.163607 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:17.164267 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:17.164386 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:17.166456 (+ 2070us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:17.166472 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:17.166474 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:17.166497 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:17.167694 (+ 1197us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:17.168157 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:17.168159 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:17.168160 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:17.168199 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:17.168431 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:17.168433 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:17.168435 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:17.168569 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:17.168641 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:17.169267 (+ 626us) server_negotiation.cc:300] Negotiation successful
0131 03:02:17.169397 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260131 03:02:17.170153 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:17.159956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:42321 (local address 127.6.22.67:34329)
0131 03:02:17.160384 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:17.160427 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:17.161245 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:17.161676 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:17.161683 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:17.162061 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:17.162937 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:17.162950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:17.163771 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:17.163774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:17.164167 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:17.164175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:17.164343 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:17.164891 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:17.164913 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:17.166329 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:17.167839 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:17.167844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:17.167851 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:17.168069 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:17.168276 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:17.168279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:17.168280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:17.168357 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:17.168707 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:17.168715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:17.168955 (+ 240us) client_negotiation.cc:769] Sending connection context
0131 03:02:17.169220 (+ 265us) client_negotiation.cc:240] Negotiation successful
0131 03:02:17.169462 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":96,"threads_started":1}
I20260131 03:02:17.171260 9164 heartbeater.cc:344] Connected to a master server at 127.6.22.126:42321
I20260131 03:02:17.171548 9164 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:17.172224 9164 heartbeater.cc:507] Master 127.6.22.126:42321 requested a full tablet report, sending...
I20260131 03:02:17.173211 8699 ts_manager.cc:194] Registered new tserver with Master: 2b6513ffe306430c8f3bd201a31e3837 (127.6.22.67:36503)
I20260131 03:02:17.173715 8699 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:34329
I20260131 03:02:17.184180 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260131 03:02:17.196179 8738 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:17.194870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.6.22.126:42321)
0131 03:02:17.195019 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:17.195024 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:17.195162 (+ 138us) server_negotiation.cc:408] Connection header received
0131 03:02:17.195277 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:17.195279 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:17.195325 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:17.195405 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:17.196050 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51038: BlockingRecv error: recv got EOF from 127.0.0.1:51038 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260131 03:02:17.197438 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 8763
I20260131 03:02:17.203436 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 8899
I20260131 03:02:17.208508 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 9035
I20260131 03:02:17.213616 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 8667
2026-01-31T03:02:17Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3137 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:02:17 dist-test-slave-j368 krb5kdc[9176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:02:17 dist-test-slave-j368 krb5kdc[9176](info): set up 2 sockets
Jan 31 03:02:17 dist-test-slave-j368 krb5kdc[9176](info): commencing operation
krb5kdc: starting...
W20260131 03:02:19.198506 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.963s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:02:19 dist-test-slave-j368 krb5kdc[9176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:02:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:19.331784 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:37295
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:37295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:19.436478 9192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:19.436805 9192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:19.436926 9192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:19.440131 9192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:19.440218 9192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:19.440270 9192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:19.440331 9192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:19.440373 9192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:19.444484 9192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:37295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:37295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:19.445551 9192 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:19.446390 9192 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:19.452430 9197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:19.452431 9198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:19.452644 9192 server_base.cc:1060] running on GCE node
W20260131 03:02:19.452432 9200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:19.453332 9192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:19.454375 9192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:19.455590 9192 hybrid_clock.cc:648] HybridClock initialized: now 1769828539455579 us; error 38 us; skew 500 ppm
Jan 31 03:02:19 dist-test-slave-j368 krb5kdc[9176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828539, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:19.458390 9192 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:02:19.459409 9192 webserver.cc:492] Webserver started at http://127.6.22.126:46813/ using document root <none> and password file <none>
I20260131 03:02:19.459975 9192 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:19.460050 9192 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:19.460253 9192 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:19.462013 9192 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "6dddc52ff7a848f49123461b5d923271"
format_stamp: "Formatted at 2026-01-31 03:02:19 on dist-test-slave-j368"
server_key: "d063ffe94362e99ae28847a51e30a9c8"
server_key_iv: "d78fe8b7ee96aad0f55eb93dc7217d1e"
server_key_version: "encryptionkey@0"
I20260131 03:02:19.462494 9192 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "6dddc52ff7a848f49123461b5d923271"
format_stamp: "Formatted at 2026-01-31 03:02:19 on dist-test-slave-j368"
server_key: "d063ffe94362e99ae28847a51e30a9c8"
server_key_iv: "d78fe8b7ee96aad0f55eb93dc7217d1e"
server_key_version: "encryptionkey@0"
I20260131 03:02:19.465762 9192 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260131 03:02:19.467892 9207 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:19.468905 9192 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260131 03:02:19.469036 9192 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "6dddc52ff7a848f49123461b5d923271"
format_stamp: "Formatted at 2026-01-31 03:02:19 on dist-test-slave-j368"
server_key: "d063ffe94362e99ae28847a51e30a9c8"
server_key_iv: "d78fe8b7ee96aad0f55eb93dc7217d1e"
server_key_version: "encryptionkey@0"
I20260131 03:02:19.469132 9192 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:19.486909 9192 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:19.489511 9192 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:19.489714 9192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:19.496958 9192 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:37295
I20260131 03:02:19.496955 9259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:37295 every 8 connection(s)
I20260131 03:02:19.497912 9192 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:19.500602 9260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:19.505870 9260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271: Bootstrap starting.
I20260131 03:02:19.507094 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 9192
I20260131 03:02:19.507244 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:19.507462 6233 external_mini_cluster.cc:1468] Setting key fa49d5c36948c3b0c8a26d8f341a83e2
I20260131 03:02:19.508261 9260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:19.509097 9260 log.cc:826] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:19.511055 9260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271: No bootstrap required, opened a new log
I20260131 03:02:19.513892 9260 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dddc52ff7a848f49123461b5d923271" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 37295 } }
I20260131 03:02:19.514084 9260 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:19.514173 9260 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dddc52ff7a848f49123461b5d923271, State: Initialized, Role: FOLLOWER
I20260131 03:02:19.514616 9260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dddc52ff7a848f49123461b5d923271" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 37295 } }
Jan 31 03:02:19 dist-test-slave-j368 krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:19.514755 9260 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:19.514837 9260 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:19.514950 9260 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:19.516144 9260 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dddc52ff7a848f49123461b5d923271" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 37295 } }
I20260131 03:02:19.516512 9260 leader_election.cc:304] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dddc52ff7a848f49123461b5d923271; no voters:
I20260131 03:02:19.516883 9260 leader_election.cc:290] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:19.516954 9265 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:19.517148 9265 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [term 1 LEADER]: Becoming Leader. State: Replica: 6dddc52ff7a848f49123461b5d923271, State: Running, Role: LEADER
I20260131 03:02:19.517411 9265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dddc52ff7a848f49123461b5d923271" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 37295 } }
I20260131 03:02:19.518007 9260 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:19.518510 9267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dddc52ff7a848f49123461b5d923271. Latest consensus state: current_term: 1 leader_uuid: "6dddc52ff7a848f49123461b5d923271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dddc52ff7a848f49123461b5d923271" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 37295 } } }
I20260131 03:02:19.518633 9267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:19.519191 9266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dddc52ff7a848f49123461b5d923271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dddc52ff7a848f49123461b5d923271" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 37295 } } }
I20260131 03:02:19.519297 9266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:19.519572 9269 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:19.521100 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:19.508463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57524 (local address 127.6.22.126:37295)
0131 03:02:19.508882 (+ 419us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:19.508895 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:19.508935 (+ 40us) server_negotiation.cc:408] Connection header received
0131 03:02:19.509621 (+ 686us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:19.509641 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:19.509954 (+ 313us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:19.510265 (+ 311us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:19.511039 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:19.512132 (+ 1093us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:19.512913 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:19.513183 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:19.515303 (+ 2120us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:19.515323 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:19.515335 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:19.515377 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:19.517598 (+ 2221us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:19.518021 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:19.518027 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:19.518034 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:19.518145 (+ 111us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:19.518452 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:19.518455 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:19.518457 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:19.518988 (+ 531us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:19.519185 (+ 197us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:19.520032 (+ 847us) server_negotiation.cc:300] Negotiation successful
0131 03:02:19.520344 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":139,"threads_started":1}
I20260131 03:02:19.523861 9269 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:19.528837 9269 catalog_manager.cc:1357] Generated new cluster ID: 76acc10d35e94583beb43acad6695a3e
I20260131 03:02:19.528905 9269 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:19.542923 9269 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:19.544098 9269 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:19.553232 9269 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 6dddc52ff7a848f49123461b5d923271: Generated new TSK 0
I20260131 03:02:19.553974 9269 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:19.597689 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:37295
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:19.707017 9288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:19.707221 9288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:19.707276 9288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:19.710443 9288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:19.710495 9288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:19.710589 9288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:02:19.714615 9288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:37295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:19.715648 9288 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:19.716460 9288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:02:19.722750 9288 server_base.cc:1060] running on GCE node
W20260131 03:02:19.722894 9294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:19.722925 9293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:19.722911 9296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:19.723315 9288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:19.723841 9288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:19.725028 9288 hybrid_clock.cc:648] HybridClock initialized: now 1769828539725015 us; error 28 us; skew 500 ppm
Jan 31 03:02:19 dist-test-slave-j368 krb5kdc[9176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828539, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:19.727355 9288 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:02:19.728345 9288 webserver.cc:492] Webserver started at http://127.6.22.65:45505/ using document root <none> and password file <none>
I20260131 03:02:19.728945 9288 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:19.729028 9288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:19.729228 9288 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:19.730814 9288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "8cdc6a9bc07f49668860a5c095e35008"
format_stamp: "Formatted at 2026-01-31 03:02:19 on dist-test-slave-j368"
server_key: "245e39053df90e1983c8d26593039489"
server_key_iv: "860a06697ca7b66414cf53569a7a90c7"
server_key_version: "encryptionkey@0"
I20260131 03:02:19.731312 9288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "8cdc6a9bc07f49668860a5c095e35008"
format_stamp: "Formatted at 2026-01-31 03:02:19 on dist-test-slave-j368"
server_key: "245e39053df90e1983c8d26593039489"
server_key_iv: "860a06697ca7b66414cf53569a7a90c7"
server_key_version: "encryptionkey@0"
I20260131 03:02:19.734536 9288 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260131 03:02:19.736557 9303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:19.737524 9288 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260131 03:02:19.737689 9288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "8cdc6a9bc07f49668860a5c095e35008"
format_stamp: "Formatted at 2026-01-31 03:02:19 on dist-test-slave-j368"
server_key: "245e39053df90e1983c8d26593039489"
server_key_iv: "860a06697ca7b66414cf53569a7a90c7"
server_key_version: "encryptionkey@0"
I20260131 03:02:19.737787 9288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:19.755040 9288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:19.758205 9288 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:19.758358 9288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:19.758821 9288 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:19.759601 9288 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:19.759649 9288 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:19.759696 9288 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:19.759718 9288 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:19.768524 9288 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:36261
I20260131 03:02:19.768584 9416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:36261 every 8 connection(s)
I20260131 03:02:19.769466 9288 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:02:19.773255 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 9288
I20260131 03:02:19.773362 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:02:19.773605 6233 external_mini_cluster.cc:1468] Setting key 0e74132f17d32433a9e2f84fb929bea3
Jan 31 03:02:19 dist-test-slave-j368 krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828539, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
I20260131 03:02:19.782729 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:19.771112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:51091 (local address 127.6.22.126:37295)
0131 03:02:19.771247 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:19.771251 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:19.771829 (+ 578us) server_negotiation.cc:408] Connection header received
0131 03:02:19.772677 (+ 848us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:19.772682 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:19.772746 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:19.772859 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:19.774897 (+ 2038us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:19.775666 (+ 769us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:19.776614 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:19.776894 (+ 280us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:19.779047 (+ 2153us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:19.779074 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:19.779080 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:19.779116 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:19.780778 (+ 1662us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:19.781240 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:19.781244 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:19.781246 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:19.781301 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:19.781579 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:19.781586 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:19.781590 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:19.781811 (+ 221us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:19.781929 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:19.782393 (+ 464us) server_negotiation.cc:300] Negotiation successful
0131 03:02:19.782521 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260131 03:02:19.783244 9419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:19.771289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:37295 (local address 127.6.22.65:51091)
0131 03:02:19.771693 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:19.771727 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:19.772450 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:19.773062 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:19.773073 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:19.773577 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:19.774745 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:19.774761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:19.775821 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:19.775828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:19.776467 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:19.776480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:19.776744 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:19.777382 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:19.777407 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:19.778895 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:19.780892 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:19.780896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:19.780905 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:19.781131 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:19.781400 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:19.781402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:19.781403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:19.781481 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:19.781921 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:19.781926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:19.782148 (+ 222us) client_negotiation.cc:769] Sending connection context
0131 03:02:19.782310 (+ 162us) client_negotiation.cc:240] Negotiation successful
0131 03:02:19.782506 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":109,"threads_started":1}
I20260131 03:02:19.784552 9417 heartbeater.cc:344] Connected to a master server at 127.6.22.126:37295
I20260131 03:02:19.784894 9417 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
I20260131 03:02:19.785553 9417 heartbeater.cc:507] Master 127.6.22.126:37295 requested a full tablet report, sending...
I20260131 03:02:19.787184 9222 ts_manager.cc:194] Registered new tserver with Master: 8cdc6a9bc07f49668860a5c095e35008 (127.6.22.65:36261)
I20260131 03:02:19.788468 9222 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:51091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:19.817708 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:37295
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:19.920835 9424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:19.921082 9424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:19.921170 9424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:19.924384 9424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:19.924482 9424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:19.924594 9424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:02:19.928723 9424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:37295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:19.929791 9424 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:19.930634 9424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:19.937021 9430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:19.936975 9429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:19.937129 9424 server_base.cc:1060] running on GCE node
W20260131 03:02:19.936977 9432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:19.937577 9424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:19.938112 9424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:19.939303 9424 hybrid_clock.cc:648] HybridClock initialized: now 1769828539939293 us; error 40 us; skew 500 ppm
Jan 31 03:02:19 dist-test-slave-j368 krb5kdc[9176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828539, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:19.942081 9424 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:02:19.943153 9424 webserver.cc:492] Webserver started at http://127.6.22.66:42725/ using document root <none> and password file <none>
I20260131 03:02:19.943652 9424 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:19.943701 9424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:19.943852 9424 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:19.945541 9424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "dddafc3407124f9f895b666a58d2400f"
format_stamp: "Formatted at 2026-01-31 03:02:19 on dist-test-slave-j368"
server_key: "649c300310322e6cf252ce018f295b64"
server_key_iv: "ff2420edaffe74a67a199c9b8d3e141e"
server_key_version: "encryptionkey@0"
I20260131 03:02:19.945963 9424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "dddafc3407124f9f895b666a58d2400f"
format_stamp: "Formatted at 2026-01-31 03:02:19 on dist-test-slave-j368"
server_key: "649c300310322e6cf252ce018f295b64"
server_key_iv: "ff2420edaffe74a67a199c9b8d3e141e"
server_key_version: "encryptionkey@0"
I20260131 03:02:19.949194 9424 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260131 03:02:19.951180 9439 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:19.952227 9424 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:19.952361 9424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "dddafc3407124f9f895b666a58d2400f"
format_stamp: "Formatted at 2026-01-31 03:02:19 on dist-test-slave-j368"
server_key: "649c300310322e6cf252ce018f295b64"
server_key_iv: "ff2420edaffe74a67a199c9b8d3e141e"
server_key_version: "encryptionkey@0"
I20260131 03:02:19.952461 9424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:19.976038 9424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:19.979017 9424 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:19.979205 9424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:19.979795 9424 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:19.980724 9424 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:19.980772 9424 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:19.980832 9424 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:19.980865 9424 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:19.989921 9424 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:44157
I20260131 03:02:19.989960 9552 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:44157 every 8 connection(s)
I20260131 03:02:19.990803 9424 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:02:19.993342 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 9424
I20260131 03:02:19.993474 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:02:19.993710 6233 external_mini_cluster.cc:1468] Setting key 4eb61a293a180446d878e42ba503714e
Jan 31 03:02:19 dist-test-slave-j368 krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828539, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
I20260131 03:02:20.002975 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:19.992454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:39307 (local address 127.6.22.126:37295)
0131 03:02:19.992600 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:19.992604 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:19.993277 (+ 673us) server_negotiation.cc:408] Connection header received
0131 03:02:19.994197 (+ 920us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:19.994201 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:19.994258 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:19.994362 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:19.996333 (+ 1971us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:19.996902 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:19.997538 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:19.997725 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:19.999723 (+ 1998us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:19.999745 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:19.999765 (+ 20us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:19.999790 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:20.001170 (+ 1380us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:20.001710 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:20.001714 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:20.001717 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:20.001770 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:20.002051 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:20.002053 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:20.002054 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:20.002206 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:20.002279 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:20.002749 (+ 470us) server_negotiation.cc:300] Negotiation successful
0131 03:02:20.002860 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260131 03:02:20.003592 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:19.992749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:37295 (local address 127.6.22.66:39307)
0131 03:02:19.993161 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:19.993192 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:19.994011 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:19.994889 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:19.994896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:19.995256 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:19.996187 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:19.996200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:19.997029 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:19.997035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:19.997417 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:19.997424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:19.997650 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:19.998244 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:19.998268 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:19.999584 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:20.001322 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:20.001327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:20.001338 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:20.001567 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:20.001881 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:20.001883 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:20.001885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:20.001964 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:20.002288 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:20.002292 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:20.002513 (+ 221us) client_negotiation.cc:769] Sending connection context
0131 03:02:20.002705 (+ 192us) client_negotiation.cc:240] Negotiation successful
0131 03:02:20.002921 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
I20260131 03:02:20.004760 9553 heartbeater.cc:344] Connected to a master server at 127.6.22.126:37295
I20260131 03:02:20.005046 9553 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:20.005679 9553 heartbeater.cc:507] Master 127.6.22.126:37295 requested a full tablet report, sending...
I20260131 03:02:20.006870 9222 ts_manager.cc:194] Registered new tserver with Master: dddafc3407124f9f895b666a58d2400f (127.6.22.66:44157)
I20260131 03:02:20.007452 9222 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:39307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:20.040441 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:37295
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:20.146111 9560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:20.146332 9560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:20.146423 9560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:20.149696 9560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:20.149755 9560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:20.149881 9560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:02:20.153986 9560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:37295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:20.155102 9560 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:20.156028 9560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:20.162518 9568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:20.162523 9566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:20.162500 9565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:20.162709 9560 server_base.cc:1060] running on GCE node
I20260131 03:02:20.163146 9560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:20.163681 9560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:20.164827 9560 hybrid_clock.cc:648] HybridClock initialized: now 1769828540164808 us; error 47 us; skew 500 ppm
Jan 31 03:02:20 dist-test-slave-j368 krb5kdc[9176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828540, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:20.167519 9560 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:02:20.168624 9560 webserver.cc:492] Webserver started at http://127.6.22.67:38137/ using document root <none> and password file <none>
I20260131 03:02:20.169242 9560 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:20.169292 9560 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:20.169492 9560 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:20.171200 9560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "8e2d82a5a4984f1e9a1eb55ad8f4ba08"
format_stamp: "Formatted at 2026-01-31 03:02:20 on dist-test-slave-j368"
server_key: "638239fc9f620ae1dca160250939ea92"
server_key_iv: "52405a50750e84bddb13a09aa8b443bc"
server_key_version: "encryptionkey@0"
I20260131 03:02:20.171671 9560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "8e2d82a5a4984f1e9a1eb55ad8f4ba08"
format_stamp: "Formatted at 2026-01-31 03:02:20 on dist-test-slave-j368"
server_key: "638239fc9f620ae1dca160250939ea92"
server_key_iv: "52405a50750e84bddb13a09aa8b443bc"
server_key_version: "encryptionkey@0"
I20260131 03:02:20.175148 9560 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260131 03:02:20.177461 9575 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:20.178444 9560 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260131 03:02:20.178573 9560 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "8e2d82a5a4984f1e9a1eb55ad8f4ba08"
format_stamp: "Formatted at 2026-01-31 03:02:20 on dist-test-slave-j368"
server_key: "638239fc9f620ae1dca160250939ea92"
server_key_iv: "52405a50750e84bddb13a09aa8b443bc"
server_key_version: "encryptionkey@0"
I20260131 03:02:20.178682 9560 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:20.193954 9560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:20.196497 9560 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:20.196753 9560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:20.197299 9560 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:20.198108 9560 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:20.198180 9560 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:20.198246 9560 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:20.198293 9560 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:20.207033 9560 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:44455
I20260131 03:02:20.207065 9689 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:44455 every 8 connection(s)
I20260131 03:02:20.207949 9560 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:02:20 dist-test-slave-j368 krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828540, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:20.215947 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 9560
I20260131 03:02:20.216060 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:02:20.216317 6233 external_mini_cluster.cc:1468] Setting key 49a813d6b54820cbf68b4a0f2313c0b8
I20260131 03:02:20.219926 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:20.209694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:52235 (local address 127.6.22.126:37295)
0131 03:02:20.209831 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:20.209834 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:20.210389 (+ 555us) server_negotiation.cc:408] Connection header received
0131 03:02:20.211235 (+ 846us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:20.211239 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:20.211302 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:20.211402 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:20.213047 (+ 1645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:20.213514 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:20.214112 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:20.214276 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:20.216715 (+ 2439us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:20.216739 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:20.216742 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:20.216774 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:20.218297 (+ 1523us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:20.218717 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:20.218720 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:20.218721 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:20.218760 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:20.218992 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:20.218994 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:20.218995 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:20.219131 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:20.219202 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:20.219708 (+ 506us) server_negotiation.cc:300] Negotiation successful
0131 03:02:20.219805 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260131 03:02:20.220593 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:20.209860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:37295 (local address 127.6.22.67:52235)
0131 03:02:20.210251 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:20.210297 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:20.211021 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:20.211552 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:20.211559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:20.211963 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:20.212881 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:20.212894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:20.213632 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:20.213637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:20.214000 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:20.214006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:20.214217 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:20.214782 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:20.214805 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:20.216516 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:20.218392 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:20.218396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:20.218404 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:20.218619 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:20.218835 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:20.218838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:20.218839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:20.218919 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:20.219212 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:20.219217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:20.219445 (+ 228us) client_negotiation.cc:769] Sending connection context
0131 03:02:20.219641 (+ 196us) client_negotiation.cc:240] Negotiation successful
0131 03:02:20.219873 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260131 03:02:20.221652 9690 heartbeater.cc:344] Connected to a master server at 127.6.22.126:37295
I20260131 03:02:20.221908 9690 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:20.222431 9690 heartbeater.cc:507] Master 127.6.22.126:37295 requested a full tablet report, sending...
I20260131 03:02:20.223358 9222 ts_manager.cc:194] Registered new tserver with Master: 8e2d82a5a4984f1e9a1eb55ad8f4ba08 (127.6.22.67:44455)
I20260131 03:02:20.223933 9222 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:52235
I20260131 03:02:20.230057 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260131 03:02:20.358050 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:20.354298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.6.22.126:37295)
0131 03:02:20.354432 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:20.354435 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:20.355294 (+ 859us) server_negotiation.cc:408] Connection header received
0131 03:02:20.356508 (+ 1214us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:20.356512 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:20.356574 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:20.356745 (+ 171us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:20.357925 (+ 1180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58306: BlockingRecv error: recv got EOF from 127.0.0.1:58306 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260131 03:02:20.359663 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:20.358859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.6.22.126:37295)
0131 03:02:20.358978 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:20.358984 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:20.358999 (+ 15us) server_negotiation.cc:408] Connection header received
0131 03:02:20.359129 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:20.359132 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:20.359176 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:20.359256 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:20.359575 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58308: BlockingRecv error: recv got EOF from 127.0.0.1:58308 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260131 03:02:20.360743 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:20.360004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58322 (local address 127.6.22.126:37295)
0131 03:02:20.360119 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:20.360123 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:20.360135 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:02:20.360214 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:20.360217 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:20.360251 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:20.360339 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:20.360643 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58322: BlockingRecv error: recv got EOF from 127.0.0.1:58322 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260131 03:02:20.364553 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:20.363634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.6.22.126:37295)
0131 03:02:20.363736 (+ 102us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:20.363738 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:20.363749 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:02:20.363779 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:20.363781 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:20.363829 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:20.363898 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:20.364492 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58336: BlockingRecv error: recv got EOF from 127.0.0.1:58336 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260131 03:02:20.372519 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 9288
I20260131 03:02:20.378460 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 9424
I20260131 03:02:20.383819 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 9560
I20260131 03:02:20.389237 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 9192
2026-01-31T03:02:20Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3175 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:02:21 dist-test-slave-j368 krb5kdc[9713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:02:21 dist-test-slave-j368 krb5kdc[9713](info): set up 2 sockets
Jan 31 03:02:21 dist-test-slave-j368 krb5kdc[9713](info): commencing operation
krb5kdc: starting...
W20260131 03:02:23.863826 6233 mini_kdc.cc:121] Time spent starting KDC: real 3.452s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260131 03:02:24.050040 6233 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.186s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260131 03:02:24.321121 6233 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.271s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:02:24 dist-test-slave-j368 krb5kdc[9713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:02:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:24.479789 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:41963
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:41963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:24.588613 9729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:24.588835 9729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:24.588896 9729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:24.591959 9729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:24.592011 9729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:24.592036 9729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:24.592052 9729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:24.592070 9729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:24.596171 9729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:41963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:41963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:24.597244 9729 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:24.598035 9729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:24.603502 9735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:24.603588 9729 server_base.cc:1060] running on GCE node
W20260131 03:02:24.603502 9734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:24.603504 9737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:24.604266 9729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:24.605237 9729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:24.606415 9729 hybrid_clock.cc:648] HybridClock initialized: now 1769828544606393 us; error 37 us; skew 500 ppm
Jan 31 03:02:24 dist-test-slave-j368 krb5kdc[9713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:24.608937 9729 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:02:24.609932 9729 webserver.cc:492] Webserver started at http://127.6.22.126:44269/ using document root <none> and password file <none>
I20260131 03:02:24.610446 9729 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:24.610489 9729 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:24.610668 9729 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:24.612368 9729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "41a21b51746d4d86896189b359caf4eb"
format_stamp: "Formatted at 2026-01-31 03:02:24 on dist-test-slave-j368"
server_key: "978fa2a294326eb43c8306311f73fb23"
server_key_iv: "164e3bd706c630d4a1dc873d1858443e"
server_key_version: "encryptionkey@0"
I20260131 03:02:24.612874 9729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "41a21b51746d4d86896189b359caf4eb"
format_stamp: "Formatted at 2026-01-31 03:02:24 on dist-test-slave-j368"
server_key: "978fa2a294326eb43c8306311f73fb23"
server_key_iv: "164e3bd706c630d4a1dc873d1858443e"
server_key_version: "encryptionkey@0"
I20260131 03:02:24.616719 9729 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260131 03:02:24.618862 9744 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:24.619917 9729 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:24.620041 9729 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "41a21b51746d4d86896189b359caf4eb"
format_stamp: "Formatted at 2026-01-31 03:02:24 on dist-test-slave-j368"
server_key: "978fa2a294326eb43c8306311f73fb23"
server_key_iv: "164e3bd706c630d4a1dc873d1858443e"
server_key_version: "encryptionkey@0"
I20260131 03:02:24.620143 9729 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:24.641822 9729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:24.650454 9729 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:24.650619 9729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:24.658664 9796 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:41963 every 8 connection(s)
I20260131 03:02:24.658668 9729 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:41963
I20260131 03:02:24.659775 9729 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:24.662730 9797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:24.665308 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 9729
I20260131 03:02:24.665441 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:24.665671 6233 external_mini_cluster.cc:1468] Setting key bda58888be18449e16a92c1b3559d109
I20260131 03:02:24.668716 9797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb: Bootstrap starting.
I20260131 03:02:24.670802 9797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:24.671717 9797 log.cc:826] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb: Log is configured to *not* fsync() on all Append() calls
Jan 31 03:02:24 dist-test-slave-j368 krb5kdc[9713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:24.673702 9797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb: No bootstrap required, opened a new log
I20260131 03:02:24.676122 9797 raft_consensus.cc:359] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a21b51746d4d86896189b359caf4eb" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41963 } }
I20260131 03:02:24.676334 9797 raft_consensus.cc:385] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:24.676420 9797 raft_consensus.cc:740] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a21b51746d4d86896189b359caf4eb, State: Initialized, Role: FOLLOWER
I20260131 03:02:24.676937 9797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a21b51746d4d86896189b359caf4eb" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41963 } }
I20260131 03:02:24.677096 9797 raft_consensus.cc:399] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:24.677170 9797 raft_consensus.cc:493] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:24.677302 9797 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:24.678030 9800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:24.666801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.6.22.126:41963)
0131 03:02:24.667245 (+ 444us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:24.667255 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:24.667288 (+ 33us) server_negotiation.cc:408] Connection header received
0131 03:02:24.667803 (+ 515us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:24.667817 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:24.668116 (+ 299us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:24.668453 (+ 337us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:24.669372 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:24.670086 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:24.670722 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:24.671005 (+ 283us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:24.673507 (+ 2502us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:24.673524 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:24.673533 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:24.673564 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:24.675274 (+ 1710us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:24.675735 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:24.675739 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:24.675743 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:24.675812 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:24.676063 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:24.676067 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:24.676069 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:24.676449 (+ 380us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:24.676711 (+ 262us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:24.677004 (+ 293us) server_negotiation.cc:300] Negotiation successful
0131 03:02:24.677444 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":157,"threads_started":1}
I20260131 03:02:24.678151 9797 raft_consensus.cc:515] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a21b51746d4d86896189b359caf4eb" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41963 } }
I20260131 03:02:24.678552 9797 leader_election.cc:304] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41a21b51746d4d86896189b359caf4eb; no voters:
I20260131 03:02:24.678907 9797 leader_election.cc:290] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:24.679280 9802 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:24.679523 9802 raft_consensus.cc:697] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [term 1 LEADER]: Becoming Leader. State: Replica: 41a21b51746d4d86896189b359caf4eb, State: Running, Role: LEADER
I20260131 03:02:24.679894 9802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a21b51746d4d86896189b359caf4eb" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41963 } }
I20260131 03:02:24.680141 9797 sys_catalog.cc:565] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:24.680845 6233 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:02:24.683084 9803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41a21b51746d4d86896189b359caf4eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a21b51746d4d86896189b359caf4eb" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41963 } } }
I20260131 03:02:24.683192 9803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:24.683521 9804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41a21b51746d4d86896189b359caf4eb. Latest consensus state: current_term: 1 leader_uuid: "41a21b51746d4d86896189b359caf4eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a21b51746d4d86896189b359caf4eb" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41963 } } }
I20260131 03:02:24.683816 9804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb [sys.catalog]: This master's current role is: LEADER
W20260131 03:02:24.685279 9816 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:02:24.685395 9816 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:02:24.685516 9819 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:24.686228 9819 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:24.691362 9819 catalog_manager.cc:1357] Generated new cluster ID: 1a267d38de114159b07306701533bdc1
I20260131 03:02:24.691426 9819 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:24.698601 9800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:24.693284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.6.22.126:41963)
0131 03:02:24.693399 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:24.693404 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:24.693678 (+ 274us) server_negotiation.cc:408] Connection header received
0131 03:02:24.693810 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:24.693814 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:24.693867 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:24.693958 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:24.694668 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:24.695387 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:24.696281 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:24.696464 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:24.697073 (+ 609us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:24.697090 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:24.697094 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:24.698084 (+ 990us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:24.698120 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:24.698154 (+ 34us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:24.698251 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:24.698385 (+ 134us) server_negotiation.cc:300] Negotiation successful
0131 03:02:24.698456 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260131 03:02:24.699958 9819 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:24.701167 9819 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:24.708156 9819 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 41a21b51746d4d86896189b359caf4eb: Generated new TSK 0
I20260131 03:02:24.708930 9819 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:02:24.721462 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 9729
2026-01-31T03:02:24Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (4329 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:02:24 dist-test-slave-j368 krb5kdc[9830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:02:24 dist-test-slave-j368 krb5kdc[9830](info): set up 2 sockets
Jan 31 03:02:24 dist-test-slave-j368 krb5kdc[9830](info): commencing operation
krb5kdc: starting...
W20260131 03:02:26.605826 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.864s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:02:26 dist-test-slave-j368 krb5kdc[9830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:02:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:26.717283 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:33891
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:33891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:26.809419 9846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:26.809582 9846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:26.809674 9846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:26.812464 9846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:26.812513 9846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:26.812570 9846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:26.812610 9846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:26.812693 9846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:26.816274 9846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:33891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:33891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:26.817128 9846 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:26.817813 9846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:26.822527 9852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:26.822520 9854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:26.822669 9846 server_base.cc:1060] running on GCE node
W20260131 03:02:26.822780 9851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:26.823172 9846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:26.823849 9846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:26.824982 9846 hybrid_clock.cc:648] HybridClock initialized: now 1769828546824963 us; error 39 us; skew 500 ppm
Jan 31 03:02:26 dist-test-slave-j368 krb5kdc[9830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828546, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:26.827057 9846 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:02:26.827801 9846 webserver.cc:492] Webserver started at http://127.6.22.126:38963/ using document root <none> and password file <none>
I20260131 03:02:26.828227 9846 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:26.828267 9846 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:26.828389 9846 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:26.829780 9846 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "ab366c66bb6d4c7e949f97f1e04efbf7"
format_stamp: "Formatted at 2026-01-31 03:02:26 on dist-test-slave-j368"
server_key: "99857794421a2ad82f1c5a4dac21b33c"
server_key_iv: "49c216d7bdc687e816b4cd876426c58a"
server_key_version: "encryptionkey@0"
I20260131 03:02:26.830133 9846 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "ab366c66bb6d4c7e949f97f1e04efbf7"
format_stamp: "Formatted at 2026-01-31 03:02:26 on dist-test-slave-j368"
server_key: "99857794421a2ad82f1c5a4dac21b33c"
server_key_iv: "49c216d7bdc687e816b4cd876426c58a"
server_key_version: "encryptionkey@0"
I20260131 03:02:26.832858 9846 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260131 03:02:26.834633 9861 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:26.835487 9846 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:26.835562 9846 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "ab366c66bb6d4c7e949f97f1e04efbf7"
format_stamp: "Formatted at 2026-01-31 03:02:26 on dist-test-slave-j368"
server_key: "99857794421a2ad82f1c5a4dac21b33c"
server_key_iv: "49c216d7bdc687e816b4cd876426c58a"
server_key_version: "encryptionkey@0"
I20260131 03:02:26.835621 9846 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:26.867933 9846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:26.870093 9846 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:26.870210 9846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:26.876029 9846 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:33891
I20260131 03:02:26.876029 9913 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:33891 every 8 connection(s)
I20260131 03:02:26.876912 9846 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:26.879127 9914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:26.881361 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 9846
I20260131 03:02:26.881490 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:26.881695 6233 external_mini_cluster.cc:1468] Setting key b3af5dbe683000f205367067860b9916
I20260131 03:02:26.884724 9914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7: Bootstrap starting.
I20260131 03:02:26.886813 9914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:26.887444 9914 log.cc:826] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7: Log is configured to *not* fsync() on all Append() calls
Jan 31 03:02:26 dist-test-slave-j368 krb5kdc[9830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:26.889087 9914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7: No bootstrap required, opened a new log
I20260131 03:02:26.891908 9914 raft_consensus.cc:359] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab366c66bb6d4c7e949f97f1e04efbf7" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 33891 } }
I20260131 03:02:26.892107 9914 raft_consensus.cc:385] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:26.892163 9914 raft_consensus.cc:740] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab366c66bb6d4c7e949f97f1e04efbf7, State: Initialized, Role: FOLLOWER
I20260131 03:02:26.892572 9914 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab366c66bb6d4c7e949f97f1e04efbf7" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 33891 } }
I20260131 03:02:26.892671 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:26.882819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.6.22.126:33891)
0131 03:02:26.883185 (+ 366us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:26.883194 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:26.883229 (+ 35us) server_negotiation.cc:408] Connection header received
0131 03:02:26.883756 (+ 527us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:26.883775 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:26.884085 (+ 310us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:26.884385 (+ 300us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:26.885207 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:26.886189 (+ 982us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:26.886724 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:26.886958 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:26.888742 (+ 1784us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:26.888759 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:26.888769 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:26.888805 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:26.890514 (+ 1709us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:26.890844 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:26.890850 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:26.890854 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:26.890929 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:26.891095 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:26.891098 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:26.891099 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:26.891426 (+ 327us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:26.891608 (+ 182us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:26.891764 (+ 156us) server_negotiation.cc:300] Negotiation successful
0131 03:02:26.892001 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
I20260131 03:02:26.892839 9914 raft_consensus.cc:399] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:26.892897 9914 raft_consensus.cc:493] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:26.892982 9914 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:26.893635 9914 raft_consensus.cc:515] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab366c66bb6d4c7e949f97f1e04efbf7" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 33891 } }
I20260131 03:02:26.893929 9914 leader_election.cc:304] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab366c66bb6d4c7e949f97f1e04efbf7; no voters:
I20260131 03:02:26.894179 9914 leader_election.cc:290] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:26.894248 9919 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:26.894403 9919 raft_consensus.cc:697] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [term 1 LEADER]: Becoming Leader. State: Replica: ab366c66bb6d4c7e949f97f1e04efbf7, State: Running, Role: LEADER
I20260131 03:02:26.894637 9919 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab366c66bb6d4c7e949f97f1e04efbf7" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 33891 } }
I20260131 03:02:26.895084 9914 sys_catalog.cc:565] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:26.896124 9920 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab366c66bb6d4c7e949f97f1e04efbf7. Latest consensus state: current_term: 1 leader_uuid: "ab366c66bb6d4c7e949f97f1e04efbf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab366c66bb6d4c7e949f97f1e04efbf7" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 33891 } } }
I20260131 03:02:26.896240 9920 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:26.896437 9921 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab366c66bb6d4c7e949f97f1e04efbf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab366c66bb6d4c7e949f97f1e04efbf7" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 33891 } } }
I20260131 03:02:26.896525 9921 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:26.897122 9923 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:26.899609 9923 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:26.904196 9923 catalog_manager.cc:1357] Generated new cluster ID: 5e2302a655e043cb96e3a3510cb67f3e
I20260131 03:02:26.904260 9923 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:26.913388 9923 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:26.914261 9923 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:26.925284 9923 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ab366c66bb6d4c7e949f97f1e04efbf7: Generated new TSK 0
I20260131 03:02:26.925803 9923 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:26.974807 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:33891
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:27.085878 9942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:27.086045 9942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:27.086093 9942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:27.088760 9942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:27.088836 9942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:27.088932 9942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:02:27.092329 9942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:33891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:27.093267 9942 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:27.093937 9942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:27.099313 9950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:27.099349 9948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:27.099428 9942 server_base.cc:1060] running on GCE node
W20260131 03:02:27.099324 9947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:27.099851 9942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:27.100253 9942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:27.101395 9942 hybrid_clock.cc:648] HybridClock initialized: now 1769828547101374 us; error 32 us; skew 500 ppm
Jan 31 03:02:27 dist-test-slave-j368 krb5kdc[9830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828547, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:27.103636 9942 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:02:27.104429 9942 webserver.cc:492] Webserver started at http://127.6.22.65:44897/ using document root <none> and password file <none>
I20260131 03:02:27.104882 9942 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:27.104933 9942 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:27.105055 9942 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:27.106323 9942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "542d1a8b404f46bba8e342f01cbdff83"
format_stamp: "Formatted at 2026-01-31 03:02:27 on dist-test-slave-j368"
server_key: "af5030dde1ef50563d3c10a89804624c"
server_key_iv: "7d2b2159f8f89dc79d2f839cc201ee4b"
server_key_version: "encryptionkey@0"
I20260131 03:02:27.106664 9942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "542d1a8b404f46bba8e342f01cbdff83"
format_stamp: "Formatted at 2026-01-31 03:02:27 on dist-test-slave-j368"
server_key: "af5030dde1ef50563d3c10a89804624c"
server_key_iv: "7d2b2159f8f89dc79d2f839cc201ee4b"
server_key_version: "encryptionkey@0"
I20260131 03:02:27.109237 9942 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260131 03:02:27.110852 9957 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:27.111634 9942 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:27.111728 9942 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "542d1a8b404f46bba8e342f01cbdff83"
format_stamp: "Formatted at 2026-01-31 03:02:27 on dist-test-slave-j368"
server_key: "af5030dde1ef50563d3c10a89804624c"
server_key_iv: "7d2b2159f8f89dc79d2f839cc201ee4b"
server_key_version: "encryptionkey@0"
I20260131 03:02:27.111835 9942 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:27.123874 9942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:27.126087 9942 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:27.126202 9942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:27.126605 9942 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:27.127269 9942 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:27.127307 9942 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:27.127363 9942 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:27.127389 9942 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:27.134644 9942 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:41753
I20260131 03:02:27.134701 10070 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:41753 every 8 connection(s)
I20260131 03:02:27.135452 9942 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:02:27.139583 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 9942
I20260131 03:02:27.139711 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:02:27.139923 6233 external_mini_cluster.cc:1468] Setting key 857a1af7cbc57a7c17163a82b22e4866
Jan 31 03:02:27 dist-test-slave-j368 krb5kdc[9830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828547, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:27.146570 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:27.137016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:49861 (local address 127.6.22.126:33891)
0131 03:02:27.137130 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:27.137133 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:27.137624 (+ 491us) server_negotiation.cc:408] Connection header received
0131 03:02:27.138294 (+ 670us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:27.138297 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:27.138349 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:27.138436 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:27.140107 (+ 1671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.140629 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:27.141301 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.141426 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:27.143696 (+ 2270us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:27.143717 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:27.143719 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:27.143739 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:27.145003 (+ 1264us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:27.145400 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:27.145403 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:27.145405 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:27.145449 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:27.145704 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:27.145707 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:27.145709 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:27.145837 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:27.145929 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:27.146328 (+ 399us) server_negotiation.cc:300] Negotiation successful
0131 03:02:27.146438 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260131 03:02:27.147017 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:27.137196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:33891 (local address 127.6.22.65:49861)
0131 03:02:27.137517 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:27.137543 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:27.138117 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:27.138560 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:27.138566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:27.138941 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:27.139922 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:27.139938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.140774 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:27.140777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:27.141186 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:27.141196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.141382 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:27.141821 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:27.141861 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:27.143560 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:27.145110 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:27.145115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:27.145123 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:27.145309 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:27.145540 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:27.145543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:27.145544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:27.145621 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:27.145930 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:27.145936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:27.146119 (+ 183us) client_negotiation.cc:769] Sending connection context
0131 03:02:27.146285 (+ 166us) client_negotiation.cc:240] Negotiation successful
0131 03:02:27.146478 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
I20260131 03:02:27.147922 10071 heartbeater.cc:344] Connected to a master server at 127.6.22.126:33891
I20260131 03:02:27.148123 10071 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:27.148515 10071 heartbeater.cc:507] Master 127.6.22.126:33891 requested a full tablet report, sending...
I20260131 03:02:27.149760 9878 ts_manager.cc:194] Registered new tserver with Master: 542d1a8b404f46bba8e342f01cbdff83 (127.6.22.65:41753)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
I20260131 03:02:27.150767 9878 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:49861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:27.176466 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:33891
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:27.263803 10078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:27.263969 10078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:27.264015 10078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:27.266687 10078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:27.266732 10078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:27.266795 10078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:02:27.270121 10078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:33891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:27.270835 10078 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:27.271477 10078 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:27.276692 10086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:27.276714 10083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:27.276738 10084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:27.276805 10078 server_base.cc:1060] running on GCE node
I20260131 03:02:27.277169 10078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:27.277578 10078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:27.278709 10078 hybrid_clock.cc:648] HybridClock initialized: now 1769828547278700 us; error 21 us; skew 500 ppm
Jan 31 03:02:27 dist-test-slave-j368 krb5kdc[9830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828547, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:27.280813 10078 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:02:27.281579 10078 webserver.cc:492] Webserver started at http://127.6.22.66:39089/ using document root <none> and password file <none>
I20260131 03:02:27.282044 10078 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:27.282090 10078 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:27.282266 10078 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:27.283627 10078 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "c83531beaa944ed78ab2cb9bd6e33cf3"
format_stamp: "Formatted at 2026-01-31 03:02:27 on dist-test-slave-j368"
server_key: "a7d2ab0298dd2070d74495edd74cd3b5"
server_key_iv: "893e80bbb6c717bd0a673a0162aeaa30"
server_key_version: "encryptionkey@0"
I20260131 03:02:27.284026 10078 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "c83531beaa944ed78ab2cb9bd6e33cf3"
format_stamp: "Formatted at 2026-01-31 03:02:27 on dist-test-slave-j368"
server_key: "a7d2ab0298dd2070d74495edd74cd3b5"
server_key_iv: "893e80bbb6c717bd0a673a0162aeaa30"
server_key_version: "encryptionkey@0"
I20260131 03:02:27.286705 10078 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260131 03:02:27.288332 10093 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:27.289139 10078 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:27.289252 10078 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "c83531beaa944ed78ab2cb9bd6e33cf3"
format_stamp: "Formatted at 2026-01-31 03:02:27 on dist-test-slave-j368"
server_key: "a7d2ab0298dd2070d74495edd74cd3b5"
server_key_iv: "893e80bbb6c717bd0a673a0162aeaa30"
server_key_version: "encryptionkey@0"
I20260131 03:02:27.289328 10078 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:27.301582 10078 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:27.303874 10078 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:27.304024 10078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:27.304426 10078 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:27.305121 10078 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:27.305188 10078 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:27.305246 10078 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:27.305290 10078 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:27.312580 10078 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:43111
I20260131 03:02:27.312613 10206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:43111 every 8 connection(s)
I20260131 03:02:27.313360 10078 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:02:27 dist-test-slave-j368 krb5kdc[9830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828547, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:27.320350 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 10078
I20260131 03:02:27.320497 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:02:27.320796 6233 external_mini_cluster.cc:1468] Setting key 8df88128b2f70a5afd6ebfc7fd66f99f
I20260131 03:02:27.323765 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:27.314650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:36423 (local address 127.6.22.126:33891)
0131 03:02:27.314783 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:27.314786 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:27.315289 (+ 503us) server_negotiation.cc:408] Connection header received
0131 03:02:27.316008 (+ 719us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:27.316012 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:27.316065 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:27.316154 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:27.317495 (+ 1341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.317922 (+ 427us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:27.318564 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.318768 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:27.320586 (+ 1818us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:27.320606 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:27.320609 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:27.320640 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:27.322045 (+ 1405us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:27.322507 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:27.322513 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:27.322518 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:27.322572 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:27.322835 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:27.322841 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:27.322846 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:27.323047 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:27.323134 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:27.323488 (+ 354us) server_negotiation.cc:300] Negotiation successful
0131 03:02:27.323601 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:02:27.324337 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:27.314855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:33891 (local address 127.6.22.66:36423)
0131 03:02:27.315178 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:27.315204 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:27.315843 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:27.316280 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:27.316286 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:27.316560 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:27.317365 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:27.317377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.318033 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:27.318040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:27.318442 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:27.318449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.318649 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:27.319120 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:27.319142 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:27.320386 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:27.322181 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:27.322184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:27.322192 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:27.322404 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:27.322661 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:27.322663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:27.322665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:27.322749 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:27.323131 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:27.323134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:27.323295 (+ 161us) client_negotiation.cc:769] Sending connection context
0131 03:02:27.323425 (+ 130us) client_negotiation.cc:240] Negotiation successful
0131 03:02:27.323663 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20260131 03:02:27.325197 10207 heartbeater.cc:344] Connected to a master server at 127.6.22.126:33891
I20260131 03:02:27.325410 10207 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:27.325883 10207 heartbeater.cc:507] Master 127.6.22.126:33891 requested a full tablet report, sending...
I20260131 03:02:27.326768 9878 ts_manager.cc:194] Registered new tserver with Master: c83531beaa944ed78ab2cb9bd6e33cf3 (127.6.22.66:43111)
I20260131 03:02:27.327236 9878 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:36423
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:27.357187 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:33891
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:27.445166 10214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:27.445340 10214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:27.445413 10214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:27.448176 10214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:27.448249 10214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:27.448356 10214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:02:27.451644 10214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:33891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:27.452473 10214 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:27.453212 10214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:27.458115 10222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:27.458119 10220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:27.458153 10219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:27.458365 10214 server_base.cc:1060] running on GCE node
I20260131 03:02:27.458689 10214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:27.459131 10214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:27.460302 10214 hybrid_clock.cc:648] HybridClock initialized: now 1769828547460290 us; error 42 us; skew 500 ppm
Jan 31 03:02:27 dist-test-slave-j368 krb5kdc[9830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828547, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:27.462445 10214 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:02:27.463280 10214 webserver.cc:492] Webserver started at http://127.6.22.67:34375/ using document root <none> and password file <none>
I20260131 03:02:27.463701 10214 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:27.463762 10214 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:27.463920 10214 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:27.465265 10214 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "5e0be0000b9d4de285e8f80562691614"
format_stamp: "Formatted at 2026-01-31 03:02:27 on dist-test-slave-j368"
server_key: "bbfd50a144fd79867e9f86c9eae68e65"
server_key_iv: "cb8ec1a5b5b8e6f06fa54fdc43943bf9"
server_key_version: "encryptionkey@0"
I20260131 03:02:27.465646 10214 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "5e0be0000b9d4de285e8f80562691614"
format_stamp: "Formatted at 2026-01-31 03:02:27 on dist-test-slave-j368"
server_key: "bbfd50a144fd79867e9f86c9eae68e65"
server_key_iv: "cb8ec1a5b5b8e6f06fa54fdc43943bf9"
server_key_version: "encryptionkey@0"
I20260131 03:02:27.468251 10214 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260131 03:02:27.469915 10229 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:27.470697 10214 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:27.470803 10214 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "5e0be0000b9d4de285e8f80562691614"
format_stamp: "Formatted at 2026-01-31 03:02:27 on dist-test-slave-j368"
server_key: "bbfd50a144fd79867e9f86c9eae68e65"
server_key_iv: "cb8ec1a5b5b8e6f06fa54fdc43943bf9"
server_key_version: "encryptionkey@0"
I20260131 03:02:27.470882 10214 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:27.485774 10214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:27.487649 10214 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:27.487799 10214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:27.488202 10214 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:27.488896 10214 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:27.488955 10214 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:27.489007 10214 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:27.489048 10214 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:27.496594 10214 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:36815
I20260131 03:02:27.496618 10342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:36815 every 8 connection(s)
I20260131 03:02:27.497294 10214 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:02:27.500846 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 10214
I20260131 03:02:27.500952 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:02:27.501157 6233 external_mini_cluster.cc:1468] Setting key 91d77a8b6ed753ac54b5ace3c0cca44f
Jan 31 03:02:27 dist-test-slave-j368 krb5kdc[9830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828547, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:27.507233 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:27.498667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:50705 (local address 127.6.22.126:33891)
0131 03:02:27.498781 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:27.498784 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:27.499304 (+ 520us) server_negotiation.cc:408] Connection header received
0131 03:02:27.499956 (+ 652us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:27.499960 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:27.500031 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:27.500115 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:27.501589 (+ 1474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.502332 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:27.502990 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.503130 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:27.504726 (+ 1596us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:27.504738 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:27.504740 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:27.504762 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:27.505969 (+ 1207us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:27.506323 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:27.506326 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:27.506327 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:27.506364 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:27.506562 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:27.506564 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:27.506565 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:27.506670 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:27.506731 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:27.507062 (+ 331us) server_negotiation.cc:300] Negotiation successful
0131 03:02:27.507140 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260131 03:02:27.507717 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:27.498871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:33891 (local address 127.6.22.67:50705)
0131 03:02:27.499188 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:27.499214 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:27.499790 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:27.500240 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:27.500246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:27.500530 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:27.501473 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:27.501489 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.502431 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:27.502434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:27.502859 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:27.502870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.503088 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:27.503451 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:27.503470 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:27.504574 (+ 1104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:27.506062 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:27.506065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:27.506071 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:27.506249 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:27.506429 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:27.506431 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:27.506432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:27.506501 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:27.506739 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:27.506743 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:27.506894 (+ 151us) client_negotiation.cc:769] Sending connection context
0131 03:02:27.507018 (+ 124us) client_negotiation.cc:240] Negotiation successful
0131 03:02:27.507164 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260131 03:02:27.508556 10343 heartbeater.cc:344] Connected to a master server at 127.6.22.126:33891
I20260131 03:02:27.508752 10343 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:27.509120 10343 heartbeater.cc:507] Master 127.6.22.126:33891 requested a full tablet report, sending...
I20260131 03:02:27.509888 9878 ts_manager.cc:194] Registered new tserver with Master: 5e0be0000b9d4de285e8f80562691614 (127.6.22.67:36815)
I20260131 03:02:27.510347 9878 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:50705
I20260131 03:02:27.514137 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 31 03:02:27 dist-test-slave-j368 krb5kdc[9830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828547, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jan 31 03:02:27 dist-test-slave-j368 krb5kdc[9830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828547, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:27.533315 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:27.527080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36704 (local address 127.6.22.126:33891)
0131 03:02:27.527208 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:27.527213 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:27.527309 (+ 96us) server_negotiation.cc:408] Connection header received
0131 03:02:27.527430 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:27.527433 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:27.527472 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:27.527544 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:27.528202 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.528580 (+ 378us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:27.529142 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:27.529325 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:27.530848 (+ 1523us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:27.530875 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:27.530877 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:27.530895 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:27.531822 (+ 927us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:27.532215 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:27.532222 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:27.532224 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:27.532262 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:27.532495 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:27.532498 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:27.532500 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:27.532645 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:27.532842 (+ 197us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:27.533084 (+ 242us) server_negotiation.cc:300] Negotiation successful
0131 03:02:27.533173 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260131 03:02:27.533613 9878 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36704
I20260131 03:02:27.534286 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 9942
I20260131 03:02:27.539906 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 10078
I20260131 03:02:27.544996 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 10214
I20260131 03:02:27.549343 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 9846
2026-01-31T03:02:27Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2828 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:02:27 dist-test-slave-j368 krb5kdc[10355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:02:27 dist-test-slave-j368 krb5kdc[10355](info): set up 2 sockets
Jan 31 03:02:27 dist-test-slave-j368 krb5kdc[10355](info): commencing operation
krb5kdc: starting...
W20260131 03:02:29.373265 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.804s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:02:29 dist-test-slave-j368 krb5kdc[10355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:02:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:29.557267 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:45721
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:45721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:29.683310 10371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:29.683482 10371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:29.683529 10371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:29.686239 10371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:29.686288 10371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:29.686311 10371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:29.686379 10371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:29.686417 10371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:29.689874 10371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:45721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:45721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:29.690761 10371 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:29.691509 10371 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:29.696955 10379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:29.696980 10377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:29.696957 10376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:29.698544 10371 server_base.cc:1060] running on GCE node
I20260131 03:02:29.699030 10371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:29.700029 10371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:29.701288 10371 hybrid_clock.cc:648] HybridClock initialized: now 1769828549701250 us; error 64 us; skew 500 ppm
Jan 31 03:02:29 dist-test-slave-j368 krb5kdc[10355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828549, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:29.703785 10371 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:02:29.704780 10371 webserver.cc:492] Webserver started at http://127.6.22.126:37977/ using document root <none> and password file <none>
I20260131 03:02:29.705269 10371 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:29.705312 10371 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:29.705454 10371 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:29.707034 10371 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "aae3c96727c6446f903c78e04b2ae69f"
format_stamp: "Formatted at 2026-01-31 03:02:29 on dist-test-slave-j368"
server_key: "9615340f6318fb3e8ceac674ed612e7e"
server_key_iv: "397ab4b4396ae5f3b5ab49138ee8989f"
server_key_version: "encryptionkey@0"
I20260131 03:02:29.707434 10371 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "aae3c96727c6446f903c78e04b2ae69f"
format_stamp: "Formatted at 2026-01-31 03:02:29 on dist-test-slave-j368"
server_key: "9615340f6318fb3e8ceac674ed612e7e"
server_key_iv: "397ab4b4396ae5f3b5ab49138ee8989f"
server_key_version: "encryptionkey@0"
I20260131 03:02:29.710397 10371 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260131 03:02:29.712482 10386 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:29.713551 10371 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260131 03:02:29.713650 10371 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "aae3c96727c6446f903c78e04b2ae69f"
format_stamp: "Formatted at 2026-01-31 03:02:29 on dist-test-slave-j368"
server_key: "9615340f6318fb3e8ceac674ed612e7e"
server_key_iv: "397ab4b4396ae5f3b5ab49138ee8989f"
server_key_version: "encryptionkey@0"
I20260131 03:02:29.713722 10371 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:29.730712 10371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:29.736555 10371 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:29.736791 10371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:29.745738 10371 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:45721
I20260131 03:02:29.745757 10438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:45721 every 8 connection(s)
I20260131 03:02:29.746740 10371 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:29.747597 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 10371
I20260131 03:02:29.747726 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:29.747972 6233 external_mini_cluster.cc:1468] Setting key bc3f1e254932d114a6c0ec5ec74b0454
I20260131 03:02:29.750078 10439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:29.755474 10439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f: Bootstrap starting.
Jan 31 03:02:29 dist-test-slave-j368 krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:29.757575 10439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:29.758332 10439 log.cc:826] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:29.760347 10439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f: No bootstrap required, opened a new log
I20260131 03:02:29.760515 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:29.749208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.6.22.126:45721)
0131 03:02:29.749599 (+ 391us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:29.749610 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:29.749646 (+ 36us) server_negotiation.cc:408] Connection header received
0131 03:02:29.750902 (+ 1256us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:29.750924 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:29.751231 (+ 307us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:29.751530 (+ 299us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:29.752224 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:29.753449 (+ 1225us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:29.754010 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:29.754255 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:29.756206 (+ 1951us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:29.756244 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:29.756252 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:29.756284 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:29.757925 (+ 1641us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:29.758343 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:29.758347 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:29.758350 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:29.758440 (+ 90us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:29.758677 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:29.758679 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:29.758680 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:29.758975 (+ 295us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:29.759161 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:29.759563 (+ 402us) server_negotiation.cc:300] Negotiation successful
0131 03:02:29.759806 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
I20260131 03:02:29.763139 10439 raft_consensus.cc:359] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae3c96727c6446f903c78e04b2ae69f" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 45721 } }
I20260131 03:02:29.763307 10439 raft_consensus.cc:385] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:29.763342 10439 raft_consensus.cc:740] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae3c96727c6446f903c78e04b2ae69f, State: Initialized, Role: FOLLOWER
I20260131 03:02:29.763705 10439 consensus_queue.cc:260] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae3c96727c6446f903c78e04b2ae69f" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 45721 } }
I20260131 03:02:29.763814 10439 raft_consensus.cc:399] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:29.763850 10439 raft_consensus.cc:493] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:29.763916 10439 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:29.765247 10439 raft_consensus.cc:515] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae3c96727c6446f903c78e04b2ae69f" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 45721 } }
I20260131 03:02:29.765537 10439 leader_election.cc:304] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aae3c96727c6446f903c78e04b2ae69f; no voters:
I20260131 03:02:29.765785 10439 leader_election.cc:290] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:29.765921 10444 raft_consensus.cc:2804] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:29.766147 10444 raft_consensus.cc:697] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [term 1 LEADER]: Becoming Leader. State: Replica: aae3c96727c6446f903c78e04b2ae69f, State: Running, Role: LEADER
I20260131 03:02:29.766417 10444 consensus_queue.cc:237] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae3c96727c6446f903c78e04b2ae69f" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 45721 } }
I20260131 03:02:29.766821 10439 sys_catalog.cc:565] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:29.767889 10445 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aae3c96727c6446f903c78e04b2ae69f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae3c96727c6446f903c78e04b2ae69f" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 45721 } } }
I20260131 03:02:29.767995 10445 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:29.768258 10446 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [sys.catalog]: SysCatalogTable state changed. Reason: New leader aae3c96727c6446f903c78e04b2ae69f. Latest consensus state: current_term: 1 leader_uuid: "aae3c96727c6446f903c78e04b2ae69f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae3c96727c6446f903c78e04b2ae69f" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 45721 } } }
I20260131 03:02:29.768354 10446 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:29.768595 10453 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:29.771195 10453 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:29.775981 10453 catalog_manager.cc:1357] Generated new cluster ID: 00a4437b438543c990f6b9e998a2c9e4
I20260131 03:02:29.776062 10453 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:29.787896 10453 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:29.788862 10453 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:29.795395 10453 catalog_manager.cc:6027] T 00000000000000000000000000000000 P aae3c96727c6446f903c78e04b2ae69f: Generated new TSK 0
I20260131 03:02:29.796133 10453 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:29.849712 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:45721
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:29.945073 10467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:29.945299 10467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:29.945358 10467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:29.948530 10467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:29.948587 10467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:29.948689 10467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:02:29.952394 10467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:29.953442 10467 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:29.954212 10467 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:29.961654 10475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:29.961686 10473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:29.961686 10472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:29.962109 10467 server_base.cc:1060] running on GCE node
I20260131 03:02:29.962541 10467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:29.963184 10467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:29.964419 10467 hybrid_clock.cc:648] HybridClock initialized: now 1769828549964387 us; error 58 us; skew 500 ppm
Jan 31 03:02:29 dist-test-slave-j368 krb5kdc[10355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828549, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:29.968112 10467 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:02:29.969828 10467 webserver.cc:492] Webserver started at http://127.6.22.65:42865/ using document root <none> and password file <none>
I20260131 03:02:29.970366 10467 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:29.970448 10467 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:29.970654 10467 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:29.972241 10467 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "bff6deaf29604170b3e11d7ce114afa3"
format_stamp: "Formatted at 2026-01-31 03:02:29 on dist-test-slave-j368"
server_key: "71342602906f64670a0ed7344b23f181"
server_key_iv: "6432a0957d7b334e1c5ec73097fe174e"
server_key_version: "encryptionkey@0"
I20260131 03:02:29.972776 10467 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "bff6deaf29604170b3e11d7ce114afa3"
format_stamp: "Formatted at 2026-01-31 03:02:29 on dist-test-slave-j368"
server_key: "71342602906f64670a0ed7344b23f181"
server_key_iv: "6432a0957d7b334e1c5ec73097fe174e"
server_key_version: "encryptionkey@0"
I20260131 03:02:29.975615 10467 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260131 03:02:29.977660 10482 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:29.978624 10467 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:02:29.978747 10467 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "bff6deaf29604170b3e11d7ce114afa3"
format_stamp: "Formatted at 2026-01-31 03:02:29 on dist-test-slave-j368"
server_key: "71342602906f64670a0ed7344b23f181"
server_key_iv: "6432a0957d7b334e1c5ec73097fe174e"
server_key_version: "encryptionkey@0"
I20260131 03:02:29.978840 10467 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:29.999486 10467 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:30.001693 10467 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:30.001853 10467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:30.002333 10467 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:30.003159 10467 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:30.003229 10467 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:30.003290 10467 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:30.003341 10467 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:30.010991 10467 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:41777
I20260131 03:02:30.011075 10595 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:41777 every 8 connection(s)
I20260131 03:02:30.011776 10467 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:02:30.014187 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 10467
I20260131 03:02:30.014298 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:02:30.014530 6233 external_mini_cluster.cc:1468] Setting key 5b1e0c28ba454e4d2024fd1e6109dbab
Jan 31 03:02:30 dist-test-slave-j368 krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828549, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
I20260131 03:02:30.023967 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:30.013570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:42077 (local address 127.6.22.126:45721)
0131 03:02:30.013733 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:30.013737 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:30.014522 (+ 785us) server_negotiation.cc:408] Connection header received
0131 03:02:30.015588 (+ 1066us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:30.015591 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:30.015660 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:30.015749 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:30.017437 (+ 1688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.017981 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:30.018541 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.018698 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:30.020892 (+ 2194us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:30.020907 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:30.020909 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:30.020928 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:30.022140 (+ 1212us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:30.022678 (+ 538us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:30.022681 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:30.022703 (+ 22us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:30.022761 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:30.023079 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:30.023081 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:30.023082 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:30.023207 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:30.023299 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:30.023672 (+ 373us) server_negotiation.cc:300] Negotiation successful
0131 03:02:30.023787 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260131 03:02:30.024395 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:30.013818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:45721 (local address 127.6.22.65:42077)
0131 03:02:30.014361 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:30.014424 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:30.015411 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:30.015973 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:30.015981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:30.016441 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:30.017305 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:30.017316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.018085 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:30.018090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:30.018438 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:30.018444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.018632 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:30.019180 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:30.019200 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:30.020755 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:30.022263 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:30.022272 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:30.022282 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:30.022578 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:30.022851 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:30.022855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:30.022857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:30.022978 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:30.023318 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:30.023322 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:30.023497 (+ 175us) client_negotiation.cc:769] Sending connection context
0131 03:02:30.023638 (+ 141us) client_negotiation.cc:240] Negotiation successful
0131 03:02:30.023845 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":135,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
I20260131 03:02:30.025394 10596 heartbeater.cc:344] Connected to a master server at 127.6.22.126:45721
I20260131 03:02:30.025586 10596 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:30.025985 10596 heartbeater.cc:507] Master 127.6.22.126:45721 requested a full tablet report, sending...
I20260131 03:02:30.027153 10403 ts_manager.cc:194] Registered new tserver with Master: bff6deaf29604170b3e11d7ce114afa3 (127.6.22.65:41777)
I20260131 03:02:30.028100 10403 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:42077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:30.052970 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:45721
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:30.141110 10603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:30.141297 10603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:30.141376 10603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:30.144197 10603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:30.144272 10603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:30.144368 10603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:02:30.147708 10603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:30.148643 10603 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:30.149395 10603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:30.154891 10609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:30.154922 10603 server_base.cc:1060] running on GCE node
W20260131 03:02:30.154835 10608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:30.154831 10611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:30.155262 10603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:30.155735 10603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:30.156890 10603 hybrid_clock.cc:648] HybridClock initialized: now 1769828550156877 us; error 26 us; skew 500 ppm
Jan 31 03:02:30 dist-test-slave-j368 krb5kdc[10355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828550, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:30.159022 10603 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:02:30.159847 10603 webserver.cc:492] Webserver started at http://127.6.22.66:39625/ using document root <none> and password file <none>
I20260131 03:02:30.160295 10603 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:30.160337 10603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:30.160506 10603 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:30.161893 10603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "08a0321f39e54d9bb1ef1f8a46cdec1c"
format_stamp: "Formatted at 2026-01-31 03:02:30 on dist-test-slave-j368"
server_key: "4ad5f37ba3eae23e8261ac0e199dab84"
server_key_iv: "d75bf5cff0ae77145878ed640e8a31cf"
server_key_version: "encryptionkey@0"
I20260131 03:02:30.162364 10603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "08a0321f39e54d9bb1ef1f8a46cdec1c"
format_stamp: "Formatted at 2026-01-31 03:02:30 on dist-test-slave-j368"
server_key: "4ad5f37ba3eae23e8261ac0e199dab84"
server_key_iv: "d75bf5cff0ae77145878ed640e8a31cf"
server_key_version: "encryptionkey@0"
I20260131 03:02:30.165132 10603 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:02:30.166886 10618 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:30.167785 10603 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:02:30.167877 10603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "08a0321f39e54d9bb1ef1f8a46cdec1c"
format_stamp: "Formatted at 2026-01-31 03:02:30 on dist-test-slave-j368"
server_key: "4ad5f37ba3eae23e8261ac0e199dab84"
server_key_iv: "d75bf5cff0ae77145878ed640e8a31cf"
server_key_version: "encryptionkey@0"
I20260131 03:02:30.167954 10603 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:30.181350 10603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:30.183852 10603 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:30.184015 10603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:30.184453 10603 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:30.185173 10603 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:30.185214 10603 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:30.185273 10603 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:30.185307 10603 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:30.193235 10603 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:35825
I20260131 03:02:30.193256 10731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:35825 every 8 connection(s)
I20260131 03:02:30.193976 10603 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:02:30.196847 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 10603
I20260131 03:02:30.196983 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:02:30.197216 6233 external_mini_cluster.cc:1468] Setting key 60ffd95189c0c814a84b862433b781ae
Jan 31 03:02:30 dist-test-slave-j368 krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828550, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:30.204862 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:30.195337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:34951 (local address 127.6.22.126:45721)
0131 03:02:30.195485 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:30.195489 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:30.196020 (+ 531us) server_negotiation.cc:408] Connection header received
0131 03:02:30.196788 (+ 768us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:30.196791 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:30.196832 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:30.196895 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:30.198547 (+ 1652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.198973 (+ 426us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:30.199730 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.199898 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:30.201796 (+ 1898us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:30.201818 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:30.201821 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:30.201841 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:30.203205 (+ 1364us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:30.203655 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:30.203658 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:30.203660 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:30.203705 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:30.203948 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:30.203951 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:30.203953 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:30.204082 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:30.204173 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:30.204586 (+ 413us) server_negotiation.cc:300] Negotiation successful
0131 03:02:30.204723 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
I20260131 03:02:30.205343 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:30.195556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:45721 (local address 127.6.22.66:34951)
0131 03:02:30.195894 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:30.195930 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:30.196546 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:30.197060 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:30.197069 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:30.197485 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:30.198411 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:30.198423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.199084 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:30.199090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:30.199623 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:30.199635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.199824 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:30.200415 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:30.200435 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:30.201668 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:30.203343 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:30.203348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:30.203357 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:30.203561 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:30.203786 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:30.203789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:30.203790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:30.203865 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:30.204190 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:30.204196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:30.204375 (+ 179us) client_negotiation.cc:769] Sending connection context
0131 03:02:30.204543 (+ 168us) client_negotiation.cc:240] Negotiation successful
0131 03:02:30.204767 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
I20260131 03:02:30.206226 10732 heartbeater.cc:344] Connected to a master server at 127.6.22.126:45721
I20260131 03:02:30.206427 10732 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:30.206872 10732 heartbeater.cc:507] Master 127.6.22.126:45721 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
I20260131 03:02:30.207819 10403 ts_manager.cc:194] Registered new tserver with Master: 08a0321f39e54d9bb1ef1f8a46cdec1c (127.6.22.66:35825)
I20260131 03:02:30.208398 10403 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:34951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:30.234982 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:45721
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:30.325502 10739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:30.325649 10739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:30.325692 10739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:30.328234 10739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:30.328275 10739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:30.328338 10739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:02:30.331709 10739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:30.332553 10739 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:30.333312 10739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:30.338541 10747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:30.338653 10739 server_base.cc:1060] running on GCE node
W20260131 03:02:30.338553 10744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:30.338745 10745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:30.339001 10739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:30.339455 10739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:30.340595 10739 hybrid_clock.cc:648] HybridClock initialized: now 1769828550340585 us; error 21 us; skew 500 ppm
Jan 31 03:02:30 dist-test-slave-j368 krb5kdc[10355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828550, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:30.342840 10739 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:02:30.343709 10739 webserver.cc:492] Webserver started at http://127.6.22.67:37617/ using document root <none> and password file <none>
I20260131 03:02:30.344161 10739 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:30.344224 10739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:30.344393 10739 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:30.345768 10739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "1bf55e02836243bdbff9ec2728b5cc11"
format_stamp: "Formatted at 2026-01-31 03:02:30 on dist-test-slave-j368"
server_key: "f2cf8d7948e2fb3944329468c88bd5e4"
server_key_iv: "8181df3f3e7a31249711a16290bb8c32"
server_key_version: "encryptionkey@0"
I20260131 03:02:30.346163 10739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "1bf55e02836243bdbff9ec2728b5cc11"
format_stamp: "Formatted at 2026-01-31 03:02:30 on dist-test-slave-j368"
server_key: "f2cf8d7948e2fb3944329468c88bd5e4"
server_key_iv: "8181df3f3e7a31249711a16290bb8c32"
server_key_version: "encryptionkey@0"
I20260131 03:02:30.348822 10739 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260131 03:02:30.350486 10754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:30.351246 10739 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:02:30.351362 10739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "1bf55e02836243bdbff9ec2728b5cc11"
format_stamp: "Formatted at 2026-01-31 03:02:30 on dist-test-slave-j368"
server_key: "f2cf8d7948e2fb3944329468c88bd5e4"
server_key_iv: "8181df3f3e7a31249711a16290bb8c32"
server_key_version: "encryptionkey@0"
I20260131 03:02:30.351441 10739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:30.361866 10739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:30.363986 10739 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:30.364141 10739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:30.364554 10739 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:30.365278 10739 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:30.365348 10739 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:30.365402 10739 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:30.365438 10739 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:30.373127 10739 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:33523
I20260131 03:02:30.373198 10867 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:33523 every 8 connection(s)
I20260131 03:02:30.373909 10739 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:02:30.379227 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 10739
I20260131 03:02:30.379354 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:02:30.379619 6233 external_mini_cluster.cc:1468] Setting key d8e5a75362c8d1136e18be42e2a1ffce
Jan 31 03:02:30 dist-test-slave-j368 krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828550, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:30.383664 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:30.375255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:34093 (local address 127.6.22.126:45721)
0131 03:02:30.375401 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:30.375406 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:30.375888 (+ 482us) server_negotiation.cc:408] Connection header received
0131 03:02:30.376514 (+ 626us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:30.376516 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:30.376559 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:30.376622 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:30.377900 (+ 1278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.378404 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:30.378969 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.379089 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:30.381047 (+ 1958us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:30.381062 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:30.381064 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:30.381089 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:30.382179 (+ 1090us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:30.382640 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:30.382643 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:30.382644 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:30.382685 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:30.382910 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:30.382912 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:30.382914 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:30.383035 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:30.383108 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:30.383492 (+ 384us) server_negotiation.cc:300] Negotiation successful
0131 03:02:30.383569 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260131 03:02:30.384157 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:30.375449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:45721 (local address 127.6.22.67:34093)
0131 03:02:30.375776 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:30.375803 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:30.376366 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:30.376778 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:30.376784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:30.377053 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:30.377778 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:30.377790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.378501 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:30.378503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:30.378880 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:30.378885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.379087 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:30.379599 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:30.379629 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:30.380936 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:30.382278 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:30.382282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:30.382289 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:30.382558 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:30.382765 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:30.382767 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:30.382769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:30.382839 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:30.383116 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:30.383121 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:30.383309 (+ 188us) client_negotiation.cc:769] Sending connection context
0131 03:02:30.383457 (+ 148us) client_negotiation.cc:240] Negotiation successful
0131 03:02:30.383607 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260131 03:02:30.384985 10868 heartbeater.cc:344] Connected to a master server at 127.6.22.126:45721
I20260131 03:02:30.385170 10868 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:30.385547 10868 heartbeater.cc:507] Master 127.6.22.126:45721 requested a full tablet report, sending...
I20260131 03:02:30.386292 10403 ts_manager.cc:194] Registered new tserver with Master: 1bf55e02836243bdbff9ec2728b5cc11 (127.6.22.67:33523)
I20260131 03:02:30.386744 10403 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:34093
I20260131 03:02:30.392712 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 31 03:02:30 dist-test-slave-j368 krb5kdc[10355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 31 03:02:30 dist-test-slave-j368 krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.65@KRBTEST.COM
I20260131 03:02:30.412496 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:30.405639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.6.22.65:41777)
0131 03:02:30.405904 (+ 265us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:30.405909 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:30.405927 (+ 18us) server_negotiation.cc:408] Connection header received
0131 03:02:30.405971 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:30.405974 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:30.406105 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:30.406203 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:30.406841 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.407426 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:30.407927 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.408092 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:30.409742 (+ 1650us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:30.409760 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:30.409768 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:30.409788 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:30.411162 (+ 1374us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:30.411559 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:30.411562 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:30.411563 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:30.411602 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:30.411761 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:30.411763 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:30.411764 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:30.411893 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:30.412015 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:30.412184 (+ 169us) server_negotiation.cc:300] Negotiation successful
0131 03:02:30.412329 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
Jan 31 03:02:30 dist-test-slave-j368 krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:30.420471 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:30.414614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43948 (local address 127.6.22.126:45721)
0131 03:02:30.414747 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:30.414750 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:30.414761 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:02:30.414830 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:30.414832 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:30.414867 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:30.414922 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:30.415446 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.415835 (+ 389us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:30.416357 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:30.416466 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:30.418225 (+ 1759us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:30.418242 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:30.418248 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:30.418267 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:30.419237 (+ 970us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:30.419562 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:30.419566 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:30.419567 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:30.419608 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:30.419817 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:30.419820 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:30.419822 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:30.419954 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:30.420072 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:30.420252 (+ 180us) server_negotiation.cc:300] Negotiation successful
0131 03:02:30.420342 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260131 03:02:30.420825 10403 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43948
I20260131 03:02:30.421319 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 10467
I20260131 03:02:30.427538 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 10603
I20260131 03:02:30.432233 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 10739
I20260131 03:02:30.436796 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 10371
2026-01-31T03:02:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2887 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:02:30 dist-test-slave-j368 krb5kdc[10880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:02:30 dist-test-slave-j368 krb5kdc[10880](info): set up 2 sockets
Jan 31 03:02:30 dist-test-slave-j368 krb5kdc[10880](info): commencing operation
krb5kdc: starting...
W20260131 03:02:32.186205 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.730s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:02:32 dist-test-slave-j368 krb5kdc[10880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:02:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:32.293550 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:43765
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:43765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:32.378350 10896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:32.378551 10896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:32.378643 10896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:32.381385 10896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:32.381456 10896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:32.381501 10896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:32.381536 10896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:32.381569 10896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:32.385064 10896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:43765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:43765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:32.385814 10896 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:32.386436 10896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:32.390618 10904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:32.390623 10902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:32.390735 10896 server_base.cc:1060] running on GCE node
W20260131 03:02:32.390686 10901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:32.391196 10896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:32.391861 10896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:32.392998 10896 hybrid_clock.cc:648] HybridClock initialized: now 1769828552392977 us; error 32 us; skew 500 ppm
Jan 31 03:02:32 dist-test-slave-j368 krb5kdc[10880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828552, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:32.395020 10896 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:02:32.395586 10896 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:32.395632 10896 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:32.395784 10896 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:32.397096 10896 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "7a5ca5a8caa64fe184b659fd431621f3"
format_stamp: "Formatted at 2026-01-31 03:02:32 on dist-test-slave-j368"
server_key: "bb6aba0a3a6ad0b2f771c1851f1e8be7"
server_key_iv: "82a23b196d2ba60b657756846878523b"
server_key_version: "encryptionkey@0"
I20260131 03:02:32.397486 10896 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "7a5ca5a8caa64fe184b659fd431621f3"
format_stamp: "Formatted at 2026-01-31 03:02:32 on dist-test-slave-j368"
server_key: "bb6aba0a3a6ad0b2f771c1851f1e8be7"
server_key_iv: "82a23b196d2ba60b657756846878523b"
server_key_version: "encryptionkey@0"
I20260131 03:02:32.399945 10896 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260131 03:02:32.401608 10910 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:32.402402 10896 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:02:32.402523 10896 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "7a5ca5a8caa64fe184b659fd431621f3"
format_stamp: "Formatted at 2026-01-31 03:02:32 on dist-test-slave-j368"
server_key: "bb6aba0a3a6ad0b2f771c1851f1e8be7"
server_key_iv: "82a23b196d2ba60b657756846878523b"
server_key_version: "encryptionkey@0"
I20260131 03:02:32.402612 10896 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:32.416332 10896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:32.418287 10896 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:32.418444 10896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:32.424798 10896 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:43765
I20260131 03:02:32.424825 10962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:43765 every 8 connection(s)
I20260131 03:02:32.425261 10896 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:32.427220 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 10896
I20260131 03:02:32.427342 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:32.427445 10963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:32.427584 6233 external_mini_cluster.cc:1468] Setting key 914090201040fa98dd5bebaf3534a1cd
I20260131 03:02:32.433179 10963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3: Bootstrap starting.
Jan 31 03:02:32 dist-test-slave-j368 krb5kdc[10880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:32.435109 10963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:32.435694 10963 log.cc:826] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:32.437175 10963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3: No bootstrap required, opened a new log
I20260131 03:02:32.438282 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:32.428543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.6.22.126:43765)
0131 03:02:32.428872 (+ 329us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:32.428878 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:32.429001 (+ 123us) server_negotiation.cc:408] Connection header received
0131 03:02:32.429545 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:32.429556 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:32.429760 (+ 204us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:32.429954 (+ 194us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:32.430926 (+ 972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:32.431536 (+ 610us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:32.432291 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:32.432475 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:32.434747 (+ 2272us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:32.434761 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:32.434768 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:32.434794 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:32.436270 (+ 1476us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:32.436634 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:32.436638 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:32.436641 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:32.436742 (+ 101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:32.436964 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:32.436967 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:32.436968 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:32.437209 (+ 241us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:32.437395 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:32.437669 (+ 274us) server_negotiation.cc:300] Negotiation successful
0131 03:02:32.437852 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":135,"threads_started":1}
I20260131 03:02:32.439349 10963 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5ca5a8caa64fe184b659fd431621f3" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 43765 } }
I20260131 03:02:32.439534 10963 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:32.439612 10963 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5ca5a8caa64fe184b659fd431621f3, State: Initialized, Role: FOLLOWER
I20260131 03:02:32.440025 10963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5ca5a8caa64fe184b659fd431621f3" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 43765 } }
I20260131 03:02:32.440162 10963 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:32.440222 10963 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:32.440321 10963 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:32.441069 10963 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5ca5a8caa64fe184b659fd431621f3" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 43765 } }
I20260131 03:02:32.441359 10963 leader_election.cc:304] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5ca5a8caa64fe184b659fd431621f3; no voters:
I20260131 03:02:32.441606 10963 leader_election.cc:290] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:32.441860 10968 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:32.442044 10968 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [term 1 LEADER]: Becoming Leader. State: Replica: 7a5ca5a8caa64fe184b659fd431621f3, State: Running, Role: LEADER
I20260131 03:02:32.442299 10968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5ca5a8caa64fe184b659fd431621f3" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 43765 } }
I20260131 03:02:32.442557 10963 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:32.443773 10970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5ca5a8caa64fe184b659fd431621f3. Latest consensus state: current_term: 1 leader_uuid: "7a5ca5a8caa64fe184b659fd431621f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5ca5a8caa64fe184b659fd431621f3" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 43765 } } }
I20260131 03:02:32.443783 10969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a5ca5a8caa64fe184b659fd431621f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5ca5a8caa64fe184b659fd431621f3" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 43765 } } }
I20260131 03:02:32.443915 10969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:32.443914 10970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:32.444571 10977 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:32.447050 10977 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:32.451164 10977 catalog_manager.cc:1357] Generated new cluster ID: 7ddd21beec0347d1ba5c449cbbf7105b
I20260131 03:02:32.451221 10977 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:32.461791 10977 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:32.462711 10977 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:32.470031 10977 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3: Generated new TSK 0
I20260131 03:02:32.470515 10977 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:32.517553 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:43765
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:32.606353 10991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:32.606508 10991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:32.606597 10991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:32.609309 10991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:32.609398 10991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:32.609529 10991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:02:32.612996 10991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:43765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:32.613843 10991 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:32.614502 10991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:32.619719 10997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:32.619768 10991 server_base.cc:1060] running on GCE node
W20260131 03:02:32.619875 10996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:32.619714 10999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:32.620159 10991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:32.620605 10991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:32.621770 10991 hybrid_clock.cc:648] HybridClock initialized: now 1769828552621750 us; error 35 us; skew 500 ppm
Jan 31 03:02:32 dist-test-slave-j368 krb5kdc[10880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828552, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:32.623982 10991 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:02:32.624516 10991 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:32.624560 10991 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:32.624727 10991 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:32.625998 10991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1"
format_stamp: "Formatted at 2026-01-31 03:02:32 on dist-test-slave-j368"
server_key: "d5ef6c801dc531fba8dd55be6bf309a3"
server_key_iv: "0a2c5bea09373a08d30fa5eea8f95717"
server_key_version: "encryptionkey@0"
I20260131 03:02:32.626348 10991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1"
format_stamp: "Formatted at 2026-01-31 03:02:32 on dist-test-slave-j368"
server_key: "d5ef6c801dc531fba8dd55be6bf309a3"
server_key_iv: "0a2c5bea09373a08d30fa5eea8f95717"
server_key_version: "encryptionkey@0"
I20260131 03:02:32.628890 10991 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260131 03:02:32.630471 11005 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:32.631312 10991 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:32.631402 10991 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1"
format_stamp: "Formatted at 2026-01-31 03:02:32 on dist-test-slave-j368"
server_key: "d5ef6c801dc531fba8dd55be6bf309a3"
server_key_iv: "0a2c5bea09373a08d30fa5eea8f95717"
server_key_version: "encryptionkey@0"
I20260131 03:02:32.631477 10991 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:32.653074 10991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:32.655349 10991 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:32.655458 10991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:32.655869 10991 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:32.656554 10991 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:32.656595 10991 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:32.656651 10991 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:32.656718 10991 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:32.664090 10991 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:39485
I20260131 03:02:32.664155 11118 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:39485 every 8 connection(s)
I20260131 03:02:32.664539 10991 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:02:32 dist-test-slave-j368 krb5kdc[10880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828552, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:32.671553 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 10991
I20260131 03:02:32.671679 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:02:32.671895 6233 external_mini_cluster.cc:1468] Setting key ffc546aa37ef1bd182f77f9441d92389
I20260131 03:02:32.674984 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:32.665860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:51543 (local address 127.6.22.126:43765)
0131 03:02:32.665973 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:32.665977 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:32.666466 (+ 489us) server_negotiation.cc:408] Connection header received
0131 03:02:32.667163 (+ 697us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:32.667166 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:32.667223 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:32.667316 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:32.668563 (+ 1247us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:32.669053 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:32.669611 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:32.669760 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:32.671449 (+ 1689us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:32.671467 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:32.671471 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:32.671499 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:32.673079 (+ 1580us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:32.673582 (+ 503us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:32.673585 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:32.673586 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:32.673633 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:32.673916 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:32.673922 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:32.673926 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:32.674117 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:32.674220 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:32.674624 (+ 404us) server_negotiation.cc:300] Negotiation successful
0131 03:02:32.674754 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:02:32.675484 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:32.666033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:43765 (local address 127.6.22.65:51543)
0131 03:02:32.666359 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:32.666386 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:32.666965 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:32.667443 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:32.667449 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:32.667718 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:32.668430 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:32.668440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:32.669162 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:32.669169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:32.669507 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:32.669513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:32.669684 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:32.670143 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:32.670164 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:32.671316 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:32.673191 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:32.673196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:32.673205 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:32.673456 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:32.673748 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:32.673750 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:32.673752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:32.673825 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:32.674215 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:32.674220 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:32.674393 (+ 173us) client_negotiation.cc:769] Sending connection context
0131 03:02:32.674547 (+ 154us) client_negotiation.cc:240] Negotiation successful
0131 03:02:32.674786 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20260131 03:02:32.676483 11119 heartbeater.cc:344] Connected to a master server at 127.6.22.126:43765
I20260131 03:02:32.676707 11119 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:32.677052 11119 heartbeater.cc:507] Master 127.6.22.126:43765 requested a full tablet report, sending...
I20260131 03:02:32.678396 10927 ts_manager.cc:194] Registered new tserver with Master: ba0e2e81515e444eaabcf9ccc0ad18f1 (127.6.22.65:39485)
I20260131 03:02:32.679816 10927 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:51543
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:32.710968 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:43765
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:32.797853 11126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:32.797992 11126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:32.798040 11126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:32.800572 11126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:32.800616 11126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:32.800693 11126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:02:32.804100 11126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:43765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:32.804800 11126 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:32.805392 11126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:32.810339 11131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:32.810336 11134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:32.810357 11132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:32.810540 11126 server_base.cc:1060] running on GCE node
I20260131 03:02:32.810815 11126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:32.811223 11126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:32.812405 11126 hybrid_clock.cc:648] HybridClock initialized: now 1769828552812395 us; error 31 us; skew 500 ppm
Jan 31 03:02:32 dist-test-slave-j368 krb5kdc[10880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828552, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:32.814568 11126 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:02:32.815112 11126 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:32.815160 11126 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:32.815280 11126 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:32.816535 11126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "01b0a32f6c254233a44bf5f23700360f"
format_stamp: "Formatted at 2026-01-31 03:02:32 on dist-test-slave-j368"
server_key: "ede5e9152640a24e17826ccb0dbd0164"
server_key_iv: "ae7d4a17640bcf8b0332f944997482ff"
server_key_version: "encryptionkey@0"
I20260131 03:02:32.816938 11126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "01b0a32f6c254233a44bf5f23700360f"
format_stamp: "Formatted at 2026-01-31 03:02:32 on dist-test-slave-j368"
server_key: "ede5e9152640a24e17826ccb0dbd0164"
server_key_iv: "ae7d4a17640bcf8b0332f944997482ff"
server_key_version: "encryptionkey@0"
I20260131 03:02:32.819535 11126 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:32.821276 11140 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:32.822070 11126 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:32.822145 11126 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "01b0a32f6c254233a44bf5f23700360f"
format_stamp: "Formatted at 2026-01-31 03:02:32 on dist-test-slave-j368"
server_key: "ede5e9152640a24e17826ccb0dbd0164"
server_key_iv: "ae7d4a17640bcf8b0332f944997482ff"
server_key_version: "encryptionkey@0"
I20260131 03:02:32.822201 11126 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:32.851207 11126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:32.853338 11126 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:32.853464 11126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:32.853842 11126 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:32.854460 11126 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:32.854499 11126 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:32.854528 11126 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:32.854542 11126 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:32.862236 11126 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:46185
I20260131 03:02:32.862259 11253 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:46185 every 8 connection(s)
I20260131 03:02:32.862731 11126 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:02:32.864864 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 11126
I20260131 03:02:32.864984 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:02:32.865242 6233 external_mini_cluster.cc:1468] Setting key c7cfc33f0c6a88643da846e127972b4e
Jan 31 03:02:32 dist-test-slave-j368 krb5kdc[10880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828552, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
I20260131 03:02:32.873939 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:32.864127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:40165 (local address 127.6.22.126:43765)
0131 03:02:32.864255 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:32.864259 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:32.864840 (+ 581us) server_negotiation.cc:408] Connection header received
0131 03:02:32.865734 (+ 894us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:32.865737 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:32.865796 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:32.865895 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:32.867254 (+ 1359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:32.867906 (+ 652us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:32.868703 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:32.868910 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:32.870485 (+ 1575us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:32.870505 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:32.870510 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:32.870546 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:32.872001 (+ 1455us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:32.872427 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:32.872433 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:32.872438 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:32.872498 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:32.872844 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:32.872851 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:32.872854 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:32.873059 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:32.873202 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:32.873623 (+ 421us) server_negotiation.cc:300] Negotiation successful
0131 03:02:32.873740 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260131 03:02:32.874262 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:32.864333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:43765 (local address 127.6.22.66:40165)
0131 03:02:32.864709 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:32.864749 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:32.865540 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:32.866026 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:32.866033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:32.866380 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:32.867113 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:32.867123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:32.868035 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:32.868040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:32.868573 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:32.868582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:32.868757 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:32.869193 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:32.869214 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:32.870349 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:32.872111 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:32.872115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:32.872121 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:32.872305 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:32.872601 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:32.872604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:32.872605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:32.872744 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:32.873185 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:32.873189 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:32.873364 (+ 175us) client_negotiation.cc:769] Sending connection context
0131 03:02:32.873504 (+ 140us) client_negotiation.cc:240] Negotiation successful
0131 03:02:32.873739 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260131 03:02:32.875234 11254 heartbeater.cc:344] Connected to a master server at 127.6.22.126:43765
I20260131 03:02:32.875455 11254 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:32.875888 11254 heartbeater.cc:507] Master 127.6.22.126:43765 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
I20260131 03:02:32.876834 10927 ts_manager.cc:194] Registered new tserver with Master: 01b0a32f6c254233a44bf5f23700360f (127.6.22.66:46185)
I20260131 03:02:32.877404 10927 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:40165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:32.903481 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:43765
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:32.990777 11261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:32.990955 11261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:32.991047 11261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:32.993932 11261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:32.994002 11261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:32.994130 11261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:02:32.997474 11261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:43765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:32.998203 11261 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:32.998847 11261 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:33.003950 11269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:33.003919 11266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:33.003948 11267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:33.004037 11261 server_base.cc:1060] running on GCE node
I20260131 03:02:33.004431 11261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:33.004895 11261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:33.006057 11261 hybrid_clock.cc:648] HybridClock initialized: now 1769828553006050 us; error 35 us; skew 500 ppm
Jan 31 03:02:33 dist-test-slave-j368 krb5kdc[10880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828553, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:33.008220 11261 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:02:33.008821 11261 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:33.008874 11261 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:33.009042 11261 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:33.010414 11261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "58c84c7ba5e14cadbcee794fc7478382"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "cafca70291598a2b882385c02888c562"
server_key_iv: "23737f1725ad31b6d313f7a3c5d5c5bc"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.010823 11261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "58c84c7ba5e14cadbcee794fc7478382"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "cafca70291598a2b882385c02888c562"
server_key_iv: "23737f1725ad31b6d313f7a3c5d5c5bc"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.013628 11261 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260131 03:02:33.015295 11275 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.016084 11261 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:02:33.016197 11261 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "58c84c7ba5e14cadbcee794fc7478382"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "cafca70291598a2b882385c02888c562"
server_key_iv: "23737f1725ad31b6d313f7a3c5d5c5bc"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.016280 11261 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:33.038806 11261 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:33.040869 11261 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:33.041041 11261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:33.041466 11261 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:33.042150 11261 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:33.042227 11261 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.042285 11261 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:33.042326 11261 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.050688 11261 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:34561
I20260131 03:02:33.050716 11388 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:34561 every 8 connection(s)
I20260131 03:02:33.051123 11261 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:02:33.057271 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 11261
I20260131 03:02:33.057400 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:02:33.057631 6233 external_mini_cluster.cc:1468] Setting key e0d68d28bb73a001a209afea02a2ef48
Jan 31 03:02:33 dist-test-slave-j368 krb5kdc[10880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828553, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:33.061480 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.052436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:39605 (local address 127.6.22.126:43765)
0131 03:02:33.052573 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.052576 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.053146 (+ 570us) server_negotiation.cc:408] Connection header received
0131 03:02:33.053909 (+ 763us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.053912 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.053972 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.054064 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.055339 (+ 1275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.055778 (+ 439us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.056364 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.056521 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.058474 (+ 1953us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.058488 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.058490 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:33.058516 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.059853 (+ 1337us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:33.060261 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:33.060265 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:33.060266 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:33.060314 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:33.060578 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:33.060581 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:33.060582 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:33.060758 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.060849 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.061231 (+ 382us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.061349 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260131 03:02:33.061911 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.052622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:43765 (local address 127.6.22.67:39605)
0131 03:02:33.053019 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.053050 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.053727 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.054199 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.054205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.054495 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:33.055201 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:33.055212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.055888 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.055893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:33.056250 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:33.056257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.056450 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.056902 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:33.056923 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:33.058281 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:33.059959 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:33.059964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:33.059972 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:33.060167 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:33.060405 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:33.060408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:33.060409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:33.060478 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:33.060859 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:33.060865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:33.061035 (+ 170us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.061198 (+ 163us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.061411 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":105,"threads_started":1}
I20260131 03:02:33.062762 11389 heartbeater.cc:344] Connected to a master server at 127.6.22.126:43765
I20260131 03:02:33.062974 11389 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:33.063383 11389 heartbeater.cc:507] Master 127.6.22.126:43765 requested a full tablet report, sending...
I20260131 03:02:33.064216 10927 ts_manager.cc:194] Registered new tserver with Master: 58c84c7ba5e14cadbcee794fc7478382 (127.6.22.67:34561)
I20260131 03:02:33.064751 10927 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:39605
I20260131 03:02:33.070781 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:02:33.077666 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.072315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.6.22.126:43765)
0131 03:02:33.072470 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.072475 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.072596 (+ 121us) server_negotiation.cc:408] Connection header received
0131 03:02:33.072781 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.072787 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.072826 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.072906 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.073647 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.074025 (+ 378us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.074555 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.074728 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.075456 (+ 728us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.075469 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.075472 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:33.075498 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.076404 (+ 906us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:33.076753 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:33.076758 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:33.076760 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:33.076799 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:33.077009 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:33.077012 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:33.077013 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:33.077140 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.077260 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.077447 (+ 187us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.077545 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260131 03:02:33.079690 10927 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:02:33.081609 10927 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:02:33.091873 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.087927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:46185 (local address 127.0.0.1:33400)
0131 03:02:33.088181 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.088216 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.088369 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.088983 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.088989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.089012 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:33.089295 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:33.089306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.090919 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.090923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:33.091472 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:33.091479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.091592 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.091633 (+ 41us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.091703 (+ 70us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.091770 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260131 03:02:33.092020 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.088882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:34561 (local address 127.0.0.1:39066)
0131 03:02:33.089606 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.089618 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.089725 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.090083 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.090086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.090102 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:33.090271 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:33.090276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.091053 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.091056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:33.091756 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:33.091765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.091837 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.091852 (+ 15us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.091892 (+ 40us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.091925 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":81,"threads_started":1}
I20260131 03:02:33.092430 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.088078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.6.22.66:46185)
0131 03:02:33.088454 (+ 376us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.088463 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.088487 (+ 24us) server_negotiation.cc:408] Connection header received
0131 03:02:33.088584 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.088593 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.088809 (+ 216us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.089454 (+ 645us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:33.089891 (+ 437us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.090823 (+ 932us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.091593 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.092062 (+ 469us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.092153 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.092203 (+ 50us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.092291 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":82,"threads_started":1}
I20260131 03:02:33.093096 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.088723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.6.22.65:39485)
0131 03:02:33.089081 (+ 358us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.089087 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.089107 (+ 20us) server_negotiation.cc:408] Connection header received
0131 03:02:33.089188 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.089194 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.089344 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.089510 (+ 166us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:33.089829 (+ 319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.090807 (+ 978us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.092151 (+ 1344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.092712 (+ 561us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.092823 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.092884 (+ 61us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.092964 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":97,"threads_started":1}
I20260131 03:02:33.093242 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.089088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.6.22.67:34561)
0131 03:02:33.089373 (+ 285us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.089380 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.089700 (+ 320us) server_negotiation.cc:408] Connection header received
0131 03:02:33.089847 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.089851 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.089972 (+ 121us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.090093 (+ 121us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:33.090367 (+ 274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.090963 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.092189 (+ 1226us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.092901 (+ 712us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.092988 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.093038 (+ 50us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.093108 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260131 03:02:33.093317 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.088550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:39485 (local address 127.0.0.1:59932)
0131 03:02:33.088777 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.088790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.088873 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.089481 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.089484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.089516 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:33.089705 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:33.089712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.090936 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.090939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:33.091697 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:33.091705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.091804 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.091820 (+ 16us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.091862 (+ 42us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.091913 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
I20260131 03:02:33.094765 11188 tablet_service.cc:1505] Processing CreateTablet for tablet 85e1d758c7064c79931457e0028e251d (DEFAULT_TABLE table=test-table [id=85d5b811c84e4ece9d3de0cc29fba23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:33.095510 11188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85e1d758c7064c79931457e0028e251d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:33.095445 11053 tablet_service.cc:1505] Processing CreateTablet for tablet 85e1d758c7064c79931457e0028e251d (DEFAULT_TABLE table=test-table [id=85d5b811c84e4ece9d3de0cc29fba23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:33.095444 11323 tablet_service.cc:1505] Processing CreateTablet for tablet 85e1d758c7064c79931457e0028e251d (DEFAULT_TABLE table=test-table [id=85d5b811c84e4ece9d3de0cc29fba23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:33.096189 11053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85e1d758c7064c79931457e0028e251d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:33.096189 11323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85e1d758c7064c79931457e0028e251d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:33.100009 11406 tablet_bootstrap.cc:492] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f: Bootstrap starting.
I20260131 03:02:33.101147 11408 tablet_bootstrap.cc:492] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1: Bootstrap starting.
I20260131 03:02:33.101428 11407 tablet_bootstrap.cc:492] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382: Bootstrap starting.
I20260131 03:02:33.101832 11406 tablet_bootstrap.cc:654] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:33.102560 11406 log.cc:826] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:33.103221 11407 tablet_bootstrap.cc:654] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:33.103930 11407 log.cc:826] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:33.104272 11406 tablet_bootstrap.cc:492] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f: No bootstrap required, opened a new log
I20260131 03:02:33.104475 11406 ts_tablet_manager.cc:1403] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260131 03:02:33.104772 11408 tablet_bootstrap.cc:654] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:33.105477 11408 log.cc:826] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:33.105638 11407 tablet_bootstrap.cc:492] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382: No bootstrap required, opened a new log
I20260131 03:02:33.105804 11407 ts_tablet_manager.cc:1403] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260131 03:02:33.106957 11408 tablet_bootstrap.cc:492] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1: No bootstrap required, opened a new log
I20260131 03:02:33.107095 11408 ts_tablet_manager.cc:1403] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260131 03:02:33.107626 11406 raft_consensus.cc:359] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b0a32f6c254233a44bf5f23700360f" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 46185 } } peers { permanent_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 39485 } } peers { permanent_uuid: "58c84c7ba5e14cadbcee794fc7478382" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34561 } }
I20260131 03:02:33.107787 11406 raft_consensus.cc:385] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:33.107851 11406 raft_consensus.cc:740] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b0a32f6c254233a44bf5f23700360f, State: Initialized, Role: FOLLOWER
I20260131 03:02:33.108223 11406 consensus_queue.cc:260] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b0a32f6c254233a44bf5f23700360f" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 46185 } } peers { permanent_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 39485 } } peers { permanent_uuid: "58c84c7ba5e14cadbcee794fc7478382" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34561 } }
I20260131 03:02:33.108750 11254 heartbeater.cc:499] Master 127.6.22.126:43765 was elected leader, sending a full tablet report...
I20260131 03:02:33.108876 11407 raft_consensus.cc:359] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b0a32f6c254233a44bf5f23700360f" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 46185 } } peers { permanent_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 39485 } } peers { permanent_uuid: "58c84c7ba5e14cadbcee794fc7478382" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34561 } }
I20260131 03:02:33.108992 11406 ts_tablet_manager.cc:1434] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260131 03:02:33.109018 11407 raft_consensus.cc:385] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:33.109062 11407 raft_consensus.cc:740] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c84c7ba5e14cadbcee794fc7478382, State: Initialized, Role: FOLLOWER
I20260131 03:02:33.109375 11407 consensus_queue.cc:260] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b0a32f6c254233a44bf5f23700360f" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 46185 } } peers { permanent_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 39485 } } peers { permanent_uuid: "58c84c7ba5e14cadbcee794fc7478382" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34561 } }
I20260131 03:02:33.110142 11407 ts_tablet_manager.cc:1434] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260131 03:02:33.110261 11408 raft_consensus.cc:359] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b0a32f6c254233a44bf5f23700360f" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 46185 } } peers { permanent_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 39485 } } peers { permanent_uuid: "58c84c7ba5e14cadbcee794fc7478382" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34561 } }
I20260131 03:02:33.110450 11389 heartbeater.cc:499] Master 127.6.22.126:43765 was elected leader, sending a full tablet report...
I20260131 03:02:33.110474 11408 raft_consensus.cc:385] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:33.110524 11408 raft_consensus.cc:740] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0e2e81515e444eaabcf9ccc0ad18f1, State: Initialized, Role: FOLLOWER
I20260131 03:02:33.110818 11408 consensus_queue.cc:260] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b0a32f6c254233a44bf5f23700360f" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 46185 } } peers { permanent_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 39485 } } peers { permanent_uuid: "58c84c7ba5e14cadbcee794fc7478382" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34561 } }
I20260131 03:02:33.111236 11119 heartbeater.cc:499] Master 127.6.22.126:43765 was elected leader, sending a full tablet report...
I20260131 03:02:33.111441 11408 ts_tablet_manager.cc:1434] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
W20260131 03:02:33.113432 11255 tablet.cc:2378] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:02:33.148032 11414 raft_consensus.cc:493] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:02:33.148125 11414 raft_consensus.cc:515] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b0a32f6c254233a44bf5f23700360f" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 46185 } } peers { permanent_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 39485 } } peers { permanent_uuid: "58c84c7ba5e14cadbcee794fc7478382" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34561 } }
I20260131 03:02:33.148844 11414 leader_election.cc:290] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01b0a32f6c254233a44bf5f23700360f (127.6.22.66:46185), 58c84c7ba5e14cadbcee794fc7478382 (127.6.22.67:34561)
I20260131 03:02:33.151206 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.148991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:46185 (local address 127.6.22.65:46079)
0131 03:02:33.149085 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.149099 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.149164 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.149363 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.149365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.149382 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:33.149578 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:33.149586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.150363 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.150367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:33.150973 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:33.150981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.151056 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.151069 (+ 13us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.151099 (+ 30us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.151135 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260131 03:02:33.151638 11415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.149207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:34561 (local address 127.6.22.65:53001)
0131 03:02:33.149483 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.149492 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.149564 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.149749 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.149752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.149767 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:33.150004 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:33.150009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.150718 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.150720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:33.151316 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:33.151324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.151422 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.151436 (+ 14us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.151478 (+ 42us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.151530 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20260131 03:02:33.151674 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.149073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:46079 (local address 127.6.22.66:46185)
0131 03:02:33.149178 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.149182 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.149196 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:02:33.149235 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.149239 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.149281 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.149351 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:33.149692 (+ 341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.150266 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.151082 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.151480 (+ 398us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.151509 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.151546 (+ 37us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.151590 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260131 03:02:33.152038 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.149253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:53001 (local address 127.6.22.67:34561)
0131 03:02:33.149360 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.149363 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.149570 (+ 207us) server_negotiation.cc:408] Connection header received
0131 03:02:33.149613 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.149618 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.149668 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.149731 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:33.150100 (+ 369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.150610 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.151430 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.151824 (+ 394us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.151851 (+ 27us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.151894 (+ 43us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.151949 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260131 03:02:33.152055 11208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85e1d758c7064c79931457e0028e251d" candidate_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01b0a32f6c254233a44bf5f23700360f" is_pre_election: true
I20260131 03:02:33.152232 11208 raft_consensus.cc:2468] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0e2e81515e444eaabcf9ccc0ad18f1 in term 0.
I20260131 03:02:33.152515 11343 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85e1d758c7064c79931457e0028e251d" candidate_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58c84c7ba5e14cadbcee794fc7478382" is_pre_election: true
I20260131 03:02:33.152734 11007 leader_election.cc:304] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b0a32f6c254233a44bf5f23700360f, ba0e2e81515e444eaabcf9ccc0ad18f1; no voters:
I20260131 03:02:33.152750 11343 raft_consensus.cc:2468] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0e2e81515e444eaabcf9ccc0ad18f1 in term 0.
I20260131 03:02:33.152951 11414 raft_consensus.cc:2804] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:02:33.153033 11414 raft_consensus.cc:493] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:02:33.153115 11414 raft_consensus.cc:3060] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:33.153900 11414 raft_consensus.cc:515] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b0a32f6c254233a44bf5f23700360f" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 46185 } } peers { permanent_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 39485 } } peers { permanent_uuid: "58c84c7ba5e14cadbcee794fc7478382" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34561 } }
I20260131 03:02:33.154170 11414 leader_election.cc:290] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [CANDIDATE]: Term 1 election: Requested vote from peers 01b0a32f6c254233a44bf5f23700360f (127.6.22.66:46185), 58c84c7ba5e14cadbcee794fc7478382 (127.6.22.67:34561)
I20260131 03:02:33.154419 11208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85e1d758c7064c79931457e0028e251d" candidate_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01b0a32f6c254233a44bf5f23700360f"
I20260131 03:02:33.154474 11343 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85e1d758c7064c79931457e0028e251d" candidate_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58c84c7ba5e14cadbcee794fc7478382"
I20260131 03:02:33.154541 11208 raft_consensus.cc:3060] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:33.154592 11343 raft_consensus.cc:3060] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:33.155278 11208 raft_consensus.cc:2468] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0e2e81515e444eaabcf9ccc0ad18f1 in term 1.
I20260131 03:02:33.155310 11343 raft_consensus.cc:2468] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0e2e81515e444eaabcf9ccc0ad18f1 in term 1.
I20260131 03:02:33.155551 11007 leader_election.cc:304] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58c84c7ba5e14cadbcee794fc7478382, ba0e2e81515e444eaabcf9ccc0ad18f1; no voters:
I20260131 03:02:33.155718 11414 raft_consensus.cc:2804] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:33.155871 11414 raft_consensus.cc:697] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [term 1 LEADER]: Becoming Leader. State: Replica: ba0e2e81515e444eaabcf9ccc0ad18f1, State: Running, Role: LEADER
I20260131 03:02:33.156060 11414 consensus_queue.cc:237] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b0a32f6c254233a44bf5f23700360f" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 46185 } } peers { permanent_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 39485 } } peers { permanent_uuid: "58c84c7ba5e14cadbcee794fc7478382" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34561 } }
I20260131 03:02:33.158273 10926 catalog_manager.cc:5654] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0e2e81515e444eaabcf9ccc0ad18f1 (127.6.22.65). New cstate: current_term: 1 leader_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b0a32f6c254233a44bf5f23700360f" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba0e2e81515e444eaabcf9ccc0ad18f1" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 39485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58c84c7ba5e14cadbcee794fc7478382" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34561 } health_report { overall_health: UNKNOWN } } }
W20260131 03:02:33.165334 11120 tablet.cc:2378] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:02:33.168573 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.166508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.6.22.65:39485)
0131 03:02:33.166604 (+ 96us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.166607 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.166617 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:02:33.166691 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.166694 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.166729 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.166789 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:33.167027 (+ 238us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.167400 (+ 373us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:33.167883 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:33.167988 (+ 105us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:33.168045 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:33.168331 (+ 286us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:33.168409 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.168479 (+ 70us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.168512 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260131 03:02:33.172219 11343 raft_consensus.cc:1275] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382 [term 1 FOLLOWER]: Refusing update from remote peer ba0e2e81515e444eaabcf9ccc0ad18f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:33.172243 11208 raft_consensus.cc:1275] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f [term 1 FOLLOWER]: Refusing update from remote peer ba0e2e81515e444eaabcf9ccc0ad18f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:33.172816 11416 consensus_queue.cc:1048] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58c84c7ba5e14cadbcee794fc7478382" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:33.172945 11414 consensus_queue.cc:1048] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01b0a32f6c254233a44bf5f23700360f" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:33.177521 11419 mvcc.cc:204] Tried to move back new op lower bound from 7249217753788928000 to 7249217753729335296. Current Snapshot: MvccSnapshot[applied={T|T < 7249217753788928000}]
I20260131 03:02:33.179864 11420 mvcc.cc:204] Tried to move back new op lower bound from 7249217753788928000 to 7249217753729335296. Current Snapshot: MvccSnapshot[applied={T|T < 7249217753788928000}]
I20260131 03:02:33.186296 10925 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:33.186388 10925 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:33.188079 10925 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 7a5ca5a8caa64fe184b659fd431621f3: Sending DeleteTablet for 3 replicas of tablet 85e1d758c7064c79931457e0028e251d
I20260131 03:02:33.188607 11188 tablet_service.cc:1552] Processing DeleteTablet for tablet 85e1d758c7064c79931457e0028e251d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:33 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:33400
I20260131 03:02:33.188860 11053 tablet_service.cc:1552] Processing DeleteTablet for tablet 85e1d758c7064c79931457e0028e251d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:33 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:59932
I20260131 03:02:33.189030 11323 tablet_service.cc:1552] Processing DeleteTablet for tablet 85e1d758c7064c79931457e0028e251d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:33 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:39066
I20260131 03:02:33.189173 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 10991
I20260131 03:02:33.189219 11430 tablet_replica.cc:333] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1: stopping tablet replica
I20260131 03:02:33.189421 11430 raft_consensus.cc:2243] T 85e1d758c7064c79931457e0028e251d P ba0e2e81515e444eaabcf9ccc0ad18f1 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:02:33.189571 11429 tablet_replica.cc:333] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f: stopping tablet replica
I20260131 03:02:33.189778 11429 raft_consensus.cc:2243] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:33.189971 11429 raft_consensus.cc:2272] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:33.191093 11429 ts_tablet_manager.cc:1916] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:33.191308 11431 tablet_replica.cc:333] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382: stopping tablet replica
I20260131 03:02:33.191558 11431 raft_consensus.cc:2243] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:33.191785 11431 raft_consensus.cc:2272] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:33.192947 11431 ts_tablet_manager.cc:1916] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:33.193117 11429 ts_tablet_manager.cc:1929] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:02:33.193215 11429 log.cc:1199] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/85e1d758c7064c79931457e0028e251d
I20260131 03:02:33.193498 11429 ts_tablet_manager.cc:1950] T 85e1d758c7064c79931457e0028e251d P 01b0a32f6c254233a44bf5f23700360f: Deleting consensus metadata
I20260131 03:02:33.194352 10912 catalog_manager.cc:4985] TS 01b0a32f6c254233a44bf5f23700360f (127.6.22.66:46185): tablet 85e1d758c7064c79931457e0028e251d (table test-table [id=85d5b811c84e4ece9d3de0cc29fba23a]) successfully deleted
I20260131 03:02:33.195509 11431 ts_tablet_manager.cc:1929] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:02:33.195595 11431 log.cc:1199] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/85e1d758c7064c79931457e0028e251d
I20260131 03:02:33.195863 11431 ts_tablet_manager.cc:1950] T 85e1d758c7064c79931457e0028e251d P 58c84c7ba5e14cadbcee794fc7478382: Deleting consensus metadata
W20260131 03:02:33.196434 10912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.22.65:39485 (error 108)
I20260131 03:02:33.196527 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 11126
I20260131 03:02:33.197214 10912 catalog_manager.cc:4985] TS 58c84c7ba5e14cadbcee794fc7478382 (127.6.22.67:34561): tablet 85e1d758c7064c79931457e0028e251d (table test-table [id=85d5b811c84e4ece9d3de0cc29fba23a]) successfully deleted
I20260131 03:02:33.197947 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.197679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:39485 (local address 127.0.0.1:59942)
0131 03:02:33.197816 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.197873 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.65:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260131 03:02:33.198076 10912 catalog_manager.cc:4712] TS ba0e2e81515e444eaabcf9ccc0ad18f1 (127.6.22.65:39485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85e1d758c7064c79931457e0028e251d: Network error: Client connection negotiation failed: client connection to 127.6.22.65:39485: connect: Connection refused (error 111)
I20260131 03:02:33.201938 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 11261
I20260131 03:02:33.206607 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 10896
2026-01-31T03:02:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2770 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-01-31T03:02:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:33Z Disabled control of system clock
I20260131 03:02:33.233150 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:43171
--webserver_interface=127.6.22.126
--webserver_port=0
--builtin_ntp_servers=127.6.22.84:42043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:43171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260131 03:02:33.320703 11436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:33.320849 11436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:33.320886 11436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:33.323680 11436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:33.323722 11436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:33.323736 11436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:33.323751 11436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:33.323765 11436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:33.326884 11436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:43171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:43171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:33.327575 11436 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:33.328365 11436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:33.332841 11442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:33.332831 11441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:33.332831 11444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:33.333096 11436 server_base.cc:1060] running on GCE node
I20260131 03:02:33.333523 11436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:33.334268 11436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:33.335431 11436 hybrid_clock.cc:648] HybridClock initialized: now 1769828553335425 us; error 32 us; skew 500 ppm
I20260131 03:02:33.336948 11436 webserver.cc:492] Webserver started at http://127.6.22.126:38323/ using document root <none> and password file <none>
I20260131 03:02:33.337386 11436 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:33.337458 11436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:33.337641 11436 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:33.338925 11436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "5f2d9e4ae3e34ff2bcc942d439aaee81"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "06f9235dd42b716678ba5f513bf2f06f"
server_key_iv: "5bc89c4492bd53a7810bc758f2958529"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.339325 11436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "5f2d9e4ae3e34ff2bcc942d439aaee81"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "06f9235dd42b716678ba5f513bf2f06f"
server_key_iv: "5bc89c4492bd53a7810bc758f2958529"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.341977 11436 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260131 03:02:33.343834 11450 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.344628 11436 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:33.344789 11436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "5f2d9e4ae3e34ff2bcc942d439aaee81"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "06f9235dd42b716678ba5f513bf2f06f"
server_key_iv: "5bc89c4492bd53a7810bc758f2958529"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.344871 11436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:33.348583 11436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:33.349014 11436 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:33.349159 11436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:33.356640 11436 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:43171
I20260131 03:02:33.356688 11502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:43171 every 8 connection(s)
I20260131 03:02:33.357496 11436 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:33.359609 11503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:33.363938 11503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81: Bootstrap starting.
I20260131 03:02:33.365672 11503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:33.366218 11503 log.cc:826] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:33.366677 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 11436
I20260131 03:02:33.366788 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:33.366991 6233 external_mini_cluster.cc:1468] Setting key 2cd30977fe015b4c5290757b11d8da45
I20260131 03:02:33.367782 11503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81: No bootstrap required, opened a new log
I20260131 03:02:33.369998 11503 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2d9e4ae3e34ff2bcc942d439aaee81" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 43171 } }
I20260131 03:02:33.370162 11503 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:33.370247 11503 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2d9e4ae3e34ff2bcc942d439aaee81, State: Initialized, Role: FOLLOWER
I20260131 03:02:33.370659 11503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2d9e4ae3e34ff2bcc942d439aaee81" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 43171 } }
I20260131 03:02:33.370795 11503 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:33.370855 11503 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:33.370937 11503 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:33.371636 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.367839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.6.22.126:43171)
0131 03:02:33.368303 (+ 464us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.368309 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.368333 (+ 24us) server_negotiation.cc:408] Connection header received
0131 03:02:33.369718 (+ 1385us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.369733 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.369925 (+ 192us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.370162 (+ 237us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.370758 (+ 596us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.370762 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.370776 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:33.370802 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.370847 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:33.370858 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.370947 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.371054 (+ 107us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.371224 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":102,"threads_started":1}
I20260131 03:02:33.371665 11503 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2d9e4ae3e34ff2bcc942d439aaee81" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 43171 } }
I20260131 03:02:33.371953 11503 leader_election.cc:304] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f2d9e4ae3e34ff2bcc942d439aaee81; no voters:
I20260131 03:02:33.372182 11503 leader_election.cc:290] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:33.372277 11508 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:33.372547 11508 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [term 1 LEADER]: Becoming Leader. State: Replica: 5f2d9e4ae3e34ff2bcc942d439aaee81, State: Running, Role: LEADER
I20260131 03:02:33.372892 11508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2d9e4ae3e34ff2bcc942d439aaee81" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 43171 } }
I20260131 03:02:33.373190 11503 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:33.373888 11508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f2d9e4ae3e34ff2bcc942d439aaee81. Latest consensus state: current_term: 1 leader_uuid: "5f2d9e4ae3e34ff2bcc942d439aaee81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2d9e4ae3e34ff2bcc942d439aaee81" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 43171 } } }
I20260131 03:02:33.374063 11508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:33.374179 11510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f2d9e4ae3e34ff2bcc942d439aaee81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2d9e4ae3e34ff2bcc942d439aaee81" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 43171 } } }
I20260131 03:02:33.374315 11510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:33.374414 11517 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:33.377187 11517 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:33.381513 11517 catalog_manager.cc:1357] Generated new cluster ID: ac748724ea4d4c50bcbc5fb9f2446734
I20260131 03:02:33.381565 11517 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:33.398413 11517 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:33.398643 11517 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:33.406351 11517 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81: Generated new TSK 0
I20260131 03:02:33.406893 11517 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:02:33.410893 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:43171
--builtin_ntp_servers=127.6.22.84:42043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260131 03:02:33.497263 11527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:33.497442 11527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:33.497514 11527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:33.500101 11527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:33.500165 11527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:33.500250 11527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:02:33.503463 11527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:43171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:33.504283 11527 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:33.505208 11527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:33.510502 11532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:33.510493 11533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:33.510672 11527 server_base.cc:1060] running on GCE node
W20260131 03:02:33.510733 11535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:33.510987 11527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:33.511445 11527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:33.512604 11527 hybrid_clock.cc:648] HybridClock initialized: now 1769828553512597 us; error 24 us; skew 500 ppm
I20260131 03:02:33.514117 11527 webserver.cc:492] Webserver started at http://127.6.22.65:44819/ using document root <none> and password file <none>
I20260131 03:02:33.514576 11527 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:33.514642 11527 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:33.514832 11527 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:33.516125 11527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "edb50e39601040778756d56aecdedef2"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "4babca25f9abec4d1b8cec3c368b5864"
server_key_iv: "0a5cab8877fc981863e64c766903d47a"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.516525 11527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "edb50e39601040778756d56aecdedef2"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "4babca25f9abec4d1b8cec3c368b5864"
server_key_iv: "0a5cab8877fc981863e64c766903d47a"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.519119 11527 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260131 03:02:33.520838 11541 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.521651 11527 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:33.521770 11527 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "edb50e39601040778756d56aecdedef2"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "4babca25f9abec4d1b8cec3c368b5864"
server_key_iv: "0a5cab8877fc981863e64c766903d47a"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.521855 11527 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:33.526028 11527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:33.526511 11527 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:33.526669 11527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:33.527093 11527 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:33.527923 11527 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:33.527987 11527 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.528048 11527 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:33.528098 11527 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.536342 11527 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:45027
I20260131 03:02:33.536376 11654 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:45027 every 8 connection(s)
I20260131 03:02:33.537145 11527 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:02:33.541208 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.538500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:60139 (local address 127.6.22.126:43171)
0131 03:02:33.538630 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.538635 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.539324 (+ 689us) server_negotiation.cc:408] Connection header received
0131 03:02:33.539899 (+ 575us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.539901 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.539935 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.540004 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.540857 (+ 853us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.540860 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.540862 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:33.540873 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.540892 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:33.540900 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.540974 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.541079 (+ 105us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.541113 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260131 03:02:33.541776 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.538744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:43171 (local address 127.6.22.65:60139)
0131 03:02:33.539217 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.539243 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.539758 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.540113 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.540118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.540182 (+ 64us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:33.540626 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:02:33.540628 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:33.540693 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:02:33.540741 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:33.540970 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:33.540975 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:33.540978 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.541106 (+ 128us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.541230 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
I20260131 03:02:33.542811 11655 heartbeater.cc:344] Connected to a master server at 127.6.22.126:43171
I20260131 03:02:33.543010 11655 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:33.543350 11655 heartbeater.cc:507] Master 127.6.22.126:43171 requested a full tablet report, sending...
I20260131 03:02:33.544548 11467 ts_manager.cc:194] Registered new tserver with Master: edb50e39601040778756d56aecdedef2 (127.6.22.65:45027)
I20260131 03:02:33.545118 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 11527
I20260131 03:02:33.545225 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:02:33.545434 6233 external_mini_cluster.cc:1468] Setting key 6181e00fd381c66731a6c6161ca1724e
I20260131 03:02:33.546955 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:43171
--builtin_ntp_servers=127.6.22.84:42043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260131 03:02:33.633777 11658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:33.633955 11658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:33.634021 11658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:33.636813 11658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:33.636875 11658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:33.636971 11658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:02:33.640033 11658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:43171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:33.640816 11658 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:33.641673 11658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:33.646935 11663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:33.646965 11664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:33.646939 11666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:33.647086 11658 server_base.cc:1060] running on GCE node
I20260131 03:02:33.647408 11658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:33.647832 11658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:33.648983 11658 hybrid_clock.cc:648] HybridClock initialized: now 1769828553648975 us; error 23 us; skew 500 ppm
I20260131 03:02:33.650301 11658 webserver.cc:492] Webserver started at http://127.6.22.66:45137/ using document root <none> and password file <none>
I20260131 03:02:33.650779 11658 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:33.650854 11658 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:33.651038 11658 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:33.652370 11658 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "ce92729663ef4789a6688d38330a40ff"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "683c9346d5dbff1ce2a9ac7c0d7b5fc6"
server_key_iv: "f5aab71e79997299d4a0f9198731342f"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.652879 11658 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "ce92729663ef4789a6688d38330a40ff"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "683c9346d5dbff1ce2a9ac7c0d7b5fc6"
server_key_iv: "f5aab71e79997299d4a0f9198731342f"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.655593 11658 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:33.657361 11672 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.658154 11658 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:33.658267 11658 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "ce92729663ef4789a6688d38330a40ff"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "683c9346d5dbff1ce2a9ac7c0d7b5fc6"
server_key_iv: "f5aab71e79997299d4a0f9198731342f"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.658349 11658 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:33.662026 11658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:33.662470 11658 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:33.662659 11658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:33.663106 11658 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:33.663925 11658 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:33.663987 11658 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.664057 11658 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:33.664103 11658 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.671545 11658 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:40451
I20260131 03:02:33.671568 11785 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:40451 every 8 connection(s)
I20260131 03:02:33.672235 11658 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:02:33.676610 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.673727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:40957 (local address 127.6.22.126:43171)
0131 03:02:33.673852 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.673855 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.674562 (+ 707us) server_negotiation.cc:408] Connection header received
0131 03:02:33.675226 (+ 664us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.675229 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.675275 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.675370 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.676193 (+ 823us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.676196 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.676199 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:33.676209 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.676230 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:33.676241 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.676329 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.676454 (+ 125us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.676494 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260131 03:02:33.677160 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.673947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:43171 (local address 127.6.22.66:40957)
0131 03:02:33.674461 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.674486 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.675037 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.675506 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.675512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.675582 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:33.675992 (+ 410us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:02:33.675993 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:33.676017 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:02:33.676057 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:33.676315 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:33.676322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:33.676325 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.676506 (+ 181us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.676721 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260131 03:02:33.677999 11786 heartbeater.cc:344] Connected to a master server at 127.6.22.126:43171
I20260131 03:02:33.678180 11786 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:33.678480 11786 heartbeater.cc:507] Master 127.6.22.126:43171 requested a full tablet report, sending...
I20260131 03:02:33.679253 11467 ts_manager.cc:194] Registered new tserver with Master: ce92729663ef4789a6688d38330a40ff (127.6.22.66:40451)
I20260131 03:02:33.681161 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 11658
I20260131 03:02:33.681272 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:02:33.681486 6233 external_mini_cluster.cc:1468] Setting key 4216b96cfff1d536c8838656275175ec
I20260131 03:02:33.682922 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:43171
--builtin_ntp_servers=127.6.22.84:42043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260131 03:02:33.771052 11789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:33.771202 11789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:33.771275 11789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:33.774253 11789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:33.774360 11789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:33.774439 11789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:02:33.777591 11789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:43171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:33.778313 11789 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:33.779135 11789 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:33.784256 11794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:33.784255 11795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:33.784300 11797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:33.784490 11789 server_base.cc:1060] running on GCE node
I20260131 03:02:33.784812 11789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:33.785192 11789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:33.786336 11789 hybrid_clock.cc:648] HybridClock initialized: now 1769828553786324 us; error 24 us; skew 500 ppm
I20260131 03:02:33.787750 11789 webserver.cc:492] Webserver started at http://127.6.22.67:33485/ using document root <none> and password file <none>
I20260131 03:02:33.788182 11789 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:33.788234 11789 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:33.788352 11789 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:33.789650 11789 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "59a7752a5245466d8107b652fbbd72b0"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "eb8a4e7937580aa6188d54ed6cb93db6"
server_key_iv: "18ebb6b778de1a0d9a952d53918e6337"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.789988 11789 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "59a7752a5245466d8107b652fbbd72b0"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "eb8a4e7937580aa6188d54ed6cb93db6"
server_key_iv: "18ebb6b778de1a0d9a952d53918e6337"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.792445 11789 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260131 03:02:33.794315 11803 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.795084 11789 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260131 03:02:33.795156 11789 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "59a7752a5245466d8107b652fbbd72b0"
format_stamp: "Formatted at 2026-01-31 03:02:33 on dist-test-slave-j368"
server_key: "eb8a4e7937580aa6188d54ed6cb93db6"
server_key_iv: "18ebb6b778de1a0d9a952d53918e6337"
server_key_version: "encryptionkey@0"
I20260131 03:02:33.795212 11789 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:33.798849 11789 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:33.799239 11789 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:33.799346 11789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:33.799706 11789 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:33.800375 11789 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:33.800415 11789 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.800444 11789 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:33.800462 11789 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:33.808040 11789 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:34281
I20260131 03:02:33.808094 11916 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:34281 every 8 connection(s)
I20260131 03:02:33.808808 11789 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:02:33.812598 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.810101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:46817 (local address 127.6.22.126:43171)
0131 03:02:33.810218 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.810221 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.810858 (+ 637us) server_negotiation.cc:408] Connection header received
0131 03:02:33.811394 (+ 536us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.811396 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.811430 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.811492 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.812239 (+ 747us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.812241 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.812243 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:33.812253 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.812271 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:33.812277 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.812345 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.812447 (+ 102us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.812496 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:02:33.813174 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.810299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:43171 (local address 127.6.22.67:46817)
0131 03:02:33.810752 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.810777 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.811257 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.811607 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.811612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.811673 (+ 61us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:33.812066 (+ 393us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:02:33.812067 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:33.812089 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:02:33.812127 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:33.812342 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:33.812347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:33.812350 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.812493 (+ 143us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.812634 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260131 03:02:33.814082 11917 heartbeater.cc:344] Connected to a master server at 127.6.22.126:43171
I20260131 03:02:33.814275 11917 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:33.814568 11917 heartbeater.cc:507] Master 127.6.22.126:43171 requested a full tablet report, sending...
I20260131 03:02:33.815341 11467 ts_manager.cc:194] Registered new tserver with Master: 59a7752a5245466d8107b652fbbd72b0 (127.6.22.67:34281)
I20260131 03:02:33.817734 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 11789
I20260131 03:02:33.817840 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:02:33.818064 6233 external_mini_cluster.cc:1468] Setting key c1a064531d72208c32a77ec74693179c
I20260131 03:02:33.819625 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:02:33.822618 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.821187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.6.22.126:43171)
0131 03:02:33.821317 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.821321 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.821436 (+ 115us) server_negotiation.cc:408] Connection header received
0131 03:02:33.821559 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.821568 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.821597 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.821674 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.822251 (+ 577us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.822255 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.822257 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:33.822267 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.822284 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:33.822291 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.822377 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.822451 (+ 74us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.822503 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260131 03:02:33.824579 11467 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:02:33.826552 11467 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:02:33.835548 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.833417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:45027 (local address 127.0.0.1:40370)
0131 03:02:33.834079 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.834089 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.834173 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.834524 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.834526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.834539 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:33.834915 (+ 376us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:02:33.834917 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:33.834929 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:02:33.834950 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:33.835213 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:33.835223 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:33.835229 (+ 6us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.835361 (+ 132us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.835418 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":97,"threads_started":1}
I20260131 03:02:33.835589 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.833568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.6.22.65:45027)
0131 03:02:33.833920 (+ 352us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.833926 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.834175 (+ 249us) server_negotiation.cc:408] Connection header received
0131 03:02:33.834265 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.834270 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.834401 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.834516 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.835033 (+ 517us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.835039 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.835049 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:33.835060 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.835101 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:33.835113 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.835233 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.835374 (+ 141us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.835437 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260131 03:02:33.836407 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.832813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:40451 (local address 127.0.0.1:36214)
0131 03:02:33.833106 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.833138 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.833224 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.833939 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.833945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.833992 (+ 47us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:33.834430 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:02:33.834431 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:33.834453 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:02:33.834515 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:33.836239 (+ 1724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:33.836241 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:33.836243 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.836289 (+ 46us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.836313 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260131 03:02:33.837945 11589 tablet_service.cc:1505] Processing CreateTablet for tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98 (DEFAULT_TABLE table=test-table [id=677edb33ea644a08beed3c4c96e0dc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:33.838599 11589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:33.840492 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.833417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:34281 (local address 127.0.0.1:46222)
0131 03:02:33.834341 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.834352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.834401 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.835526 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.835528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.835540 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:33.835893 (+ 353us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:02:33.835895 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:33.835904 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:02:33.835927 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:33.840345 (+ 4418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:33.840347 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:33.840349 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.840398 (+ 49us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.840420 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":8,"thread_start_us":87,"threads_started":1}
I20260131 03:02:33.840574 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.833765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.6.22.67:34281)
0131 03:02:33.834576 (+ 811us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.834580 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.834596 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:02:33.834629 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.834633 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.834735 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.834841 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.840204 (+ 5363us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.840208 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.840215 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:33.840223 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.840252 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:33.840261 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.840369 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.840404 (+ 35us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.840453 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":97,"threads_started":1}
I20260131 03:02:33.836469 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.832968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.6.22.66:40451)
0131 03:02:33.833525 (+ 557us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.833532 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.833555 (+ 23us) server_negotiation.cc:408] Connection header received
0131 03:02:33.833620 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.833625 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.833777 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.834646 (+ 869us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.836088 (+ 1442us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.836093 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.836100 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:33.836109 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.836147 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:33.836157 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.836257 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.836293 (+ 36us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.836343 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":118,"threads_started":1}
I20260131 03:02:33.842873 11720 tablet_service.cc:1505] Processing CreateTablet for tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98 (DEFAULT_TABLE table=test-table [id=677edb33ea644a08beed3c4c96e0dc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:33.843722 11720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:33.844225 11931 tablet_bootstrap.cc:492] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2: Bootstrap starting.
I20260131 03:02:33.844895 11851 tablet_service.cc:1505] Processing CreateTablet for tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98 (DEFAULT_TABLE table=test-table [id=677edb33ea644a08beed3c4c96e0dc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:33.845569 11851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:33.846043 11931 tablet_bootstrap.cc:654] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:33.846774 11931 log.cc:826] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:33.848802 11931 tablet_bootstrap.cc:492] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2: No bootstrap required, opened a new log
I20260131 03:02:33.848989 11931 ts_tablet_manager.cc:1403] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260131 03:02:33.850149 11934 tablet_bootstrap.cc:492] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0: Bootstrap starting.
I20260131 03:02:33.850600 11933 tablet_bootstrap.cc:492] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff: Bootstrap starting.
I20260131 03:02:33.851521 11931 raft_consensus.cc:359] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce92729663ef4789a6688d38330a40ff" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 40451 } } peers { permanent_uuid: "edb50e39601040778756d56aecdedef2" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45027 } } peers { permanent_uuid: "59a7752a5245466d8107b652fbbd72b0" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34281 } }
I20260131 03:02:33.851662 11934 tablet_bootstrap.cc:654] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:33.851687 11931 raft_consensus.cc:385] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:33.851734 11931 raft_consensus.cc:740] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb50e39601040778756d56aecdedef2, State: Initialized, Role: FOLLOWER
I20260131 03:02:33.852113 11931 consensus_queue.cc:260] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce92729663ef4789a6688d38330a40ff" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 40451 } } peers { permanent_uuid: "edb50e39601040778756d56aecdedef2" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45027 } } peers { permanent_uuid: "59a7752a5245466d8107b652fbbd72b0" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34281 } }
I20260131 03:02:33.852245 11933 tablet_bootstrap.cc:654] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:33.852348 11934 log.cc:826] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:33.852746 11655 heartbeater.cc:499] Master 127.6.22.126:43171 was elected leader, sending a full tablet report...
I20260131 03:02:33.852950 11931 ts_tablet_manager.cc:1434] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260131 03:02:33.853017 11933 log.cc:826] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:33.854656 11933 tablet_bootstrap.cc:492] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff: No bootstrap required, opened a new log
I20260131 03:02:33.854730 11934 tablet_bootstrap.cc:492] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0: No bootstrap required, opened a new log
I20260131 03:02:33.854815 11933 ts_tablet_manager.cc:1403] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260131 03:02:33.854866 11934 ts_tablet_manager.cc:1403] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260131 03:02:33.857005 11933 raft_consensus.cc:359] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce92729663ef4789a6688d38330a40ff" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 40451 } } peers { permanent_uuid: "edb50e39601040778756d56aecdedef2" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45027 } } peers { permanent_uuid: "59a7752a5245466d8107b652fbbd72b0" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34281 } }
I20260131 03:02:33.857129 11933 raft_consensus.cc:385] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:33.857156 11933 raft_consensus.cc:740] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce92729663ef4789a6688d38330a40ff, State: Initialized, Role: FOLLOWER
I20260131 03:02:33.857254 11934 raft_consensus.cc:359] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce92729663ef4789a6688d38330a40ff" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 40451 } } peers { permanent_uuid: "edb50e39601040778756d56aecdedef2" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45027 } } peers { permanent_uuid: "59a7752a5245466d8107b652fbbd72b0" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34281 } }
I20260131 03:02:33.857395 11934 raft_consensus.cc:385] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:33.857456 11934 raft_consensus.cc:740] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a7752a5245466d8107b652fbbd72b0, State: Initialized, Role: FOLLOWER
I20260131 03:02:33.857514 11933 consensus_queue.cc:260] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce92729663ef4789a6688d38330a40ff" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 40451 } } peers { permanent_uuid: "edb50e39601040778756d56aecdedef2" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45027 } } peers { permanent_uuid: "59a7752a5245466d8107b652fbbd72b0" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34281 } }
I20260131 03:02:33.857770 11934 consensus_queue.cc:260] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce92729663ef4789a6688d38330a40ff" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 40451 } } peers { permanent_uuid: "edb50e39601040778756d56aecdedef2" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45027 } } peers { permanent_uuid: "59a7752a5245466d8107b652fbbd72b0" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34281 } }
I20260131 03:02:33.858039 11786 heartbeater.cc:499] Master 127.6.22.126:43171 was elected leader, sending a full tablet report...
I20260131 03:02:33.858234 11917 heartbeater.cc:499] Master 127.6.22.126:43171 was elected leader, sending a full tablet report...
I20260131 03:02:33.858428 11934 ts_tablet_manager.cc:1434] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260131 03:02:33.858912 11933 ts_tablet_manager.cc:1434] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260131 03:02:33.923056 11787 tablet.cc:2378] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:02:33.960917 11938 raft_consensus.cc:493] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:02:33.961069 11938 raft_consensus.cc:515] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce92729663ef4789a6688d38330a40ff" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 40451 } } peers { permanent_uuid: "edb50e39601040778756d56aecdedef2" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45027 } } peers { permanent_uuid: "59a7752a5245466d8107b652fbbd72b0" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34281 } }
I20260131 03:02:33.962013 11938 leader_election.cc:290] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edb50e39601040778756d56aecdedef2 (127.6.22.65:45027), 59a7752a5245466d8107b652fbbd72b0 (127.6.22.67:34281)
I20260131 03:02:33.963755 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.962398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:35181 (local address 127.6.22.67:34281)
0131 03:02:33.962548 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.962554 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.962588 (+ 34us) server_negotiation.cc:408] Connection header received
0131 03:02:33.962629 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.962635 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.962679 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.962770 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.963371 (+ 601us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.963375 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.963378 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:33.963389 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.963408 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:33.963422 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.963510 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.963607 (+ 97us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.963645 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260131 03:02:33.963755 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.962228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:34281 (local address 127.6.22.66:35181)
0131 03:02:33.962375 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.962388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.962508 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.962804 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.962809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.962824 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:33.963232 (+ 408us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:02:33.963233 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:33.963243 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:02:33.963262 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:33.963502 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:33.963505 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:33.963508 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.963609 (+ 101us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.963643 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260131 03:02:33.964219 11871 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1a8a8ca8d4e44daaea5e7e8b2a18e98" candidate_uuid: "ce92729663ef4789a6688d38330a40ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a7752a5245466d8107b652fbbd72b0" is_pre_election: true
I20260131 03:02:33.964428 11871 raft_consensus.cc:2468] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce92729663ef4789a6688d38330a40ff in term 0.
I20260131 03:02:33.964432 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.962296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:43927 (local address 127.6.22.65:45027)
0131 03:02:33.962423 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.962428 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.963229 (+ 801us) server_negotiation.cc:408] Connection header received
0131 03:02:33.963271 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.963273 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.963306 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.963406 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.964104 (+ 698us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.964107 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.964109 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:33.964122 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.964141 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:33.964147 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.964226 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.964295 (+ 69us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.964326 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260131 03:02:33.964457 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.962560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:45027 (local address 127.6.22.66:43927)
0131 03:02:33.962902 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:33.962918 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:33.962997 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:33.963426 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:33.963430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:33.963461 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:33.963954 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:02:33.963956 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:33.963968 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:02:33.963993 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:33.964224 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:33.964227 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:33.964230 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:02:33.964290 (+ 60us) client_negotiation.cc:240] Negotiation successful
0131 03:02:33.964327 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":126,"threads_started":1}
I20260131 03:02:33.964840 11673 leader_election.cc:304] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59a7752a5245466d8107b652fbbd72b0, ce92729663ef4789a6688d38330a40ff; no voters:
I20260131 03:02:33.964910 11609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1a8a8ca8d4e44daaea5e7e8b2a18e98" candidate_uuid: "ce92729663ef4789a6688d38330a40ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb50e39601040778756d56aecdedef2" is_pre_election: true
I20260131 03:02:33.965054 11938 raft_consensus.cc:2804] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:02:33.965124 11938 raft_consensus.cc:493] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:02:33.965104 11609 raft_consensus.cc:2468] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce92729663ef4789a6688d38330a40ff in term 0.
I20260131 03:02:33.965180 11938 raft_consensus.cc:3060] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:33.966068 11938 raft_consensus.cc:515] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce92729663ef4789a6688d38330a40ff" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 40451 } } peers { permanent_uuid: "edb50e39601040778756d56aecdedef2" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45027 } } peers { permanent_uuid: "59a7752a5245466d8107b652fbbd72b0" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34281 } }
I20260131 03:02:33.966391 11938 leader_election.cc:290] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [CANDIDATE]: Term 1 election: Requested vote from peers edb50e39601040778756d56aecdedef2 (127.6.22.65:45027), 59a7752a5245466d8107b652fbbd72b0 (127.6.22.67:34281)
I20260131 03:02:33.966701 11609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1a8a8ca8d4e44daaea5e7e8b2a18e98" candidate_uuid: "ce92729663ef4789a6688d38330a40ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb50e39601040778756d56aecdedef2"
I20260131 03:02:33.966821 11609 raft_consensus.cc:3060] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:33.967937 11609 raft_consensus.cc:2468] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce92729663ef4789a6688d38330a40ff in term 1.
I20260131 03:02:33.968185 11871 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1a8a8ca8d4e44daaea5e7e8b2a18e98" candidate_uuid: "ce92729663ef4789a6688d38330a40ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a7752a5245466d8107b652fbbd72b0"
I20260131 03:02:33.968237 11674 leader_election.cc:304] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce92729663ef4789a6688d38330a40ff, edb50e39601040778756d56aecdedef2; no voters:
I20260131 03:02:33.968290 11871 raft_consensus.cc:3060] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:33.968370 11938 raft_consensus.cc:2804] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:33.968568 11938 raft_consensus.cc:697] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 1 LEADER]: Becoming Leader. State: Replica: ce92729663ef4789a6688d38330a40ff, State: Running, Role: LEADER
I20260131 03:02:33.968834 11938 consensus_queue.cc:237] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce92729663ef4789a6688d38330a40ff" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 40451 } } peers { permanent_uuid: "edb50e39601040778756d56aecdedef2" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45027 } } peers { permanent_uuid: "59a7752a5245466d8107b652fbbd72b0" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34281 } }
I20260131 03:02:33.969553 11871 raft_consensus.cc:2468] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce92729663ef4789a6688d38330a40ff in term 1.
I20260131 03:02:33.971335 11467 catalog_manager.cc:5654] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff reported cstate change: term changed from 0 to 1, leader changed from <none> to ce92729663ef4789a6688d38330a40ff (127.6.22.66). New cstate: current_term: 1 leader_uuid: "ce92729663ef4789a6688d38330a40ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce92729663ef4789a6688d38330a40ff" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 40451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edb50e39601040778756d56aecdedef2" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59a7752a5245466d8107b652fbbd72b0" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34281 } health_report { overall_health: UNKNOWN } } }
I20260131 03:02:33.983489 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:33.982137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.6.22.66:40451)
0131 03:02:33.982238 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:33.982243 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:33.982271 (+ 28us) server_negotiation.cc:408] Connection header received
0131 03:02:33.982437 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:33.982440 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:33.982496 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:33.982597 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:33.983102 (+ 505us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:33.983105 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:33.983108 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:33.983119 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:33.983138 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:33.983146 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:33.983240 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:33.983356 (+ 116us) server_negotiation.cc:300] Negotiation successful
0131 03:02:33.983389 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260131 03:02:33.987700 11609 raft_consensus.cc:1275] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P edb50e39601040778756d56aecdedef2 [term 1 FOLLOWER]: Refusing update from remote peer ce92729663ef4789a6688d38330a40ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:33.988152 11871 raft_consensus.cc:1275] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0 [term 1 FOLLOWER]: Refusing update from remote peer ce92729663ef4789a6688d38330a40ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:33.988260 11941 consensus_queue.cc:1048] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "edb50e39601040778756d56aecdedef2" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:33.988622 11938 consensus_queue.cc:1048] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "59a7752a5245466d8107b652fbbd72b0" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 34281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:33.994699 11944 mvcc.cc:204] Tried to move back new op lower bound from 7249217757128052736 to 7249217757058777088. Current Snapshot: MvccSnapshot[applied={T|T < 7249217757128052736}]
I20260131 03:02:33.995590 11945 mvcc.cc:204] Tried to move back new op lower bound from 7249217757128052736 to 7249217757058777088. Current Snapshot: MvccSnapshot[applied={T|T < 7249217757128052736}]
I20260131 03:02:33.995790 11946 mvcc.cc:204] Tried to move back new op lower bound from 7249217757128052736 to 7249217757058777088. Current Snapshot: MvccSnapshot[applied={T|T < 7249217757128052736}]
I20260131 03:02:34.001616 11467 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:34.001734 11467 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:34.003675 11467 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 5f2d9e4ae3e34ff2bcc942d439aaee81: Sending DeleteTablet for 3 replicas of tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98
I20260131 03:02:34.004251 11589 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:34 UTC) from {username='slave'} at 127.0.0.1:40370
I20260131 03:02:34.004331 11851 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:34 UTC) from {username='slave'} at 127.0.0.1:46222
I20260131 03:02:34.004693 11954 tablet_replica.cc:333] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0: stopping tablet replica
I20260131 03:02:34.004647 11720 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:34 UTC) from {username='slave'} at 127.0.0.1:36214
I20260131 03:02:34.004909 11954 raft_consensus.cc:2243] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:34.005080 11956 tablet_replica.cc:333] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff: stopping tablet replica
I20260131 03:02:34.005146 11954 raft_consensus.cc:2272] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:34.005419 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 11527
I20260131 03:02:34.005517 11956 raft_consensus.cc:2243] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:02:34.005800 11956 raft_consensus.cc:2272] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:34.006278 11954 ts_tablet_manager.cc:1916] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:34.006906 11956 ts_tablet_manager.cc:1916] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:34.008687 11954 ts_tablet_manager.cc:1929] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:02:34.008818 11954 log.cc:1199] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/f1a8a8ca8d4e44daaea5e7e8b2a18e98
I20260131 03:02:34.009142 11954 ts_tablet_manager.cc:1950] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P 59a7752a5245466d8107b652fbbd72b0: Deleting consensus metadata
I20260131 03:02:34.009456 11956 ts_tablet_manager.cc:1929] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:02:34.009581 11956 log.cc:1199] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/f1a8a8ca8d4e44daaea5e7e8b2a18e98
I20260131 03:02:34.010167 11451 catalog_manager.cc:4985] TS 59a7752a5245466d8107b652fbbd72b0 (127.6.22.67:34281): tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98 (table test-table [id=677edb33ea644a08beed3c4c96e0dc70]) successfully deleted
I20260131 03:02:34.013012 11956 ts_tablet_manager.cc:1950] T f1a8a8ca8d4e44daaea5e7e8b2a18e98 P ce92729663ef4789a6688d38330a40ff: Deleting consensus metadata
I20260131 03:02:34.014621 11451 catalog_manager.cc:4985] TS ce92729663ef4789a6688d38330a40ff (127.6.22.66:40451): tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98 (table test-table [id=677edb33ea644a08beed3c4c96e0dc70]) successfully deleted
W20260131 03:02:34.015300 11452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.22.65:45027 (error 108)
I20260131 03:02:34.015590 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 11658
I20260131 03:02:34.017233 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:34.016990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:45027 (local address 127.0.0.1:40376)
0131 03:02:34.017100 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:34.017177 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.65:45027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260131 03:02:34.017340 11452 catalog_manager.cc:4712] TS edb50e39601040778756d56aecdedef2 (127.6.22.65:45027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1a8a8ca8d4e44daaea5e7e8b2a18e98: Network error: Client connection negotiation failed: client connection to 127.6.22.65:45027: connect: Connection refused (error 111)
I20260131 03:02:34.020838 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 11789
I20260131 03:02:34.025641 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 11436
2026-01-31T03:02:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (817 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260131 03:02:34.045511 6233 mini_oidc.cc:150] Starting JWKS server
I20260131 03:02:34.046679 6233 webserver.cc:492] Webserver started at https://127.0.0.1:33323/ using document root <none> and password file <none>
I20260131 03:02:34.046839 6233 mini_oidc.cc:186] Starting OIDC Discovery server
I20260131 03:02:34.047070 6233 webserver.cc:492] Webserver started at http://127.0.0.1:43359/ using document root <none> and password file <none>
2026-01-31T03:02:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:34Z Disabled control of system clock
I20260131 03:02:34.053808 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:34809
--webserver_interface=127.6.22.126
--webserver_port=0
--builtin_ntp_servers=127.6.22.84:38851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:34809
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33323/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/testchainca.cert with env {}
W20260131 03:02:34.142241 11963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:34.142395 11963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:34.142436 11963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:34.145259 11963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:34.145303 11963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:34.145318 11963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:34.145332 11963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:34.145346 11963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260131 03:02:34.145361 11963 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260131 03:02:34.145373 11963 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33323/jwks.json
I20260131 03:02:34.148751 11963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:38851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:34809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:34809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33323/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:34.149456 11963 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:34.150058 11963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:34.154551 11971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:34.154603 11968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:34.154608 11963 server_base.cc:1060] running on GCE node
W20260131 03:02:34.154548 11969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:34.155083 11963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:34.155759 11963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:34.156889 11963 hybrid_clock.cc:648] HybridClock initialized: now 1769828554156873 us; error 29 us; skew 500 ppm
I20260131 03:02:34.158154 11963 webserver.cc:492] Webserver started at http://127.6.22.126:37853/ using document root <none> and password file <none>
I20260131 03:02:34.158600 11963 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:34.158645 11963 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:34.158820 11963 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:34.160068 11963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "a542d91cd5c14e518edb27ce05347461"
format_stamp: "Formatted at 2026-01-31 03:02:34 on dist-test-slave-j368"
server_key: "e31ffdbf75c5e89758a2fe511a7a725e"
server_key_iv: "d4507efa96be680f0a690fb7b5dbaaf8"
server_key_version: "encryptionkey@0"
I20260131 03:02:34.160449 11963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "a542d91cd5c14e518edb27ce05347461"
format_stamp: "Formatted at 2026-01-31 03:02:34 on dist-test-slave-j368"
server_key: "e31ffdbf75c5e89758a2fe511a7a725e"
server_key_iv: "d4507efa96be680f0a690fb7b5dbaaf8"
server_key_version: "encryptionkey@0"
I20260131 03:02:34.163053 11963 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:34.164806 11977 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:34.165581 11963 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:34.165689 11963 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "a542d91cd5c14e518edb27ce05347461"
format_stamp: "Formatted at 2026-01-31 03:02:34 on dist-test-slave-j368"
server_key: "e31ffdbf75c5e89758a2fe511a7a725e"
server_key_iv: "d4507efa96be680f0a690fb7b5dbaaf8"
server_key_version: "encryptionkey@0"
I20260131 03:02:34.165768 11963 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:34.186288 11963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:34.186722 11963 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:34.186872 11963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:34.192370 12029 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:34809 every 8 connection(s)
I20260131 03:02:34.192373 11963 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:34809
I20260131 03:02:34.193279 11963 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:34.195461 12030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:34.197664 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 11963
I20260131 03:02:34.197773 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:34.197989 6233 external_mini_cluster.cc:1468] Setting key c935d7955fefc2bd7288d47b30505874
I20260131 03:02:34.200242 12030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461: Bootstrap starting.
I20260131 03:02:34.201932 12030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:34.202668 12030 log.cc:826] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:34.203898 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:34.198941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.6.22.126:34809)
0131 03:02:34.199412 (+ 471us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:34.199419 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:34.199444 (+ 25us) server_negotiation.cc:408] Connection header received
0131 03:02:34.199831 (+ 387us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:34.199843 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:34.200098 (+ 255us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:34.200344 (+ 246us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:34.200790 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:34.201392 (+ 602us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:34.201950 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:34.202188 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:34.202600 (+ 412us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:34.202616 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:34.202624 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:34.202970 (+ 346us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:34.203005 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:34.203019 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:34.203125 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:34.203257 (+ 132us) server_negotiation.cc:300] Negotiation successful
0131 03:02:34.203434 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":107,"threads_started":1}
I20260131 03:02:34.204330 12030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461: No bootstrap required, opened a new log
I20260131 03:02:34.205942 6233 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:02:34.206503 12030 raft_consensus.cc:359] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a542d91cd5c14e518edb27ce05347461" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 34809 } }
I20260131 03:02:34.206687 12030 raft_consensus.cc:385] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:34.206797 12030 raft_consensus.cc:740] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a542d91cd5c14e518edb27ce05347461, State: Initialized, Role: FOLLOWER
I20260131 03:02:34.207185 12030 consensus_queue.cc:260] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a542d91cd5c14e518edb27ce05347461" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 34809 } }
I20260131 03:02:34.207307 12030 raft_consensus.cc:399] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:34.207378 12030 raft_consensus.cc:493] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:34.207471 12030 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:34.208106 12030 raft_consensus.cc:515] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a542d91cd5c14e518edb27ce05347461" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 34809 } }
I20260131 03:02:34.208467 12030 leader_election.cc:304] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a542d91cd5c14e518edb27ce05347461; no voters:
I20260131 03:02:34.208756 12030 leader_election.cc:290] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:34.208808 12035 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:34.208997 12035 raft_consensus.cc:697] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [term 1 LEADER]: Becoming Leader. State: Replica: a542d91cd5c14e518edb27ce05347461, State: Running, Role: LEADER
I20260131 03:02:34.209259 12035 consensus_queue.cc:237] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a542d91cd5c14e518edb27ce05347461" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 34809 } }
I20260131 03:02:34.209595 12030 sys_catalog.cc:565] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:34.210690 12037 sys_catalog.cc:455] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a542d91cd5c14e518edb27ce05347461. Latest consensus state: current_term: 1 leader_uuid: "a542d91cd5c14e518edb27ce05347461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a542d91cd5c14e518edb27ce05347461" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 34809 } } }
I20260131 03:02:34.210778 12037 sys_catalog.cc:458] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:34.211042 12036 sys_catalog.cc:455] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a542d91cd5c14e518edb27ce05347461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a542d91cd5c14e518edb27ce05347461" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 34809 } } }
I20260131 03:02:34.211138 12036 sys_catalog.cc:458] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461 [sys.catalog]: This master's current role is: LEADER
W20260131 03:02:34.213436 12050 catalog_manager.cc:1568] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:02:34.213497 12050 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:02:34.213572 12045 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:34.214169 12045 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:34.218089 12045 catalog_manager.cc:1357] Generated new cluster ID: 99cc20cdb1594c9890813bcd94b4f37c
I20260131 03:02:34.218140 12045 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:34.228195 12045 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:34.229180 12045 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:34.236191 12045 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a542d91cd5c14e518edb27ce05347461: Generated new TSK 0
I20260131 03:02:34.236717 12045 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:02:34.269737 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:34.249619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.6.22.126:34809)
0131 03:02:34.249736 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:34.249741 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:34.249889 (+ 148us) server_negotiation.cc:408] Connection header received
0131 03:02:34.250061 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:34.250063 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:34.250101 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:34.250164 (+ 63us) server_negotiation.cc:227] Negotiated authn=JWT
0131 03:02:34.250548 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:34.250943 (+ 395us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:34.251700 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:34.251812 (+ 112us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:34.251870 (+ 58us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0131 03:02:34.269314 (+ 17444us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0131 03:02:34.269393 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:34.269525 (+ 132us) server_negotiation.cc:300] Negotiation successful
0131 03:02:34.269567 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46,"thread_start_us":107,"threads_started":1}
I20260131 03:02:34.280169 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:34.276944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46416 (local address 127.6.22.126:34809)
0131 03:02:34.277068 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:34.277071 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:34.277173 (+ 102us) server_negotiation.cc:408] Connection header received
0131 03:02:34.277297 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:34.277300 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:34.277334 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:34.277389 (+ 55us) server_negotiation.cc:227] Negotiated authn=JWT
0131 03:02:34.277788 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:34.278229 (+ 441us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:34.279069 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:34.279212 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:34.279248 (+ 36us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0131 03:02:34.279934 (+ 686us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0131 03:02:34.280095 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46416: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":45}
W20260131 03:02:34.280246 12033 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46416: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260131 03:02:34.280977 12066 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:34.276836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:34809 (local address 127.0.0.1:46416)
0131 03:02:34.277094 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:34.277107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:34.277199 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:34.277418 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:34.277422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:34.277429 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0131 03:02:34.277687 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:34.277694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:34.278345 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:34.278349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:34.278966 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:34.278976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:34.279083 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:34.279109 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0131 03:02:34.280089 (+ 980us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0131 03:02:34.280224 (+ 135us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.22.126:34809: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
I20260131 03:02:49.289592 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:49.286353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.6.22.126:34809)
0131 03:02:49.286635 (+ 282us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:49.286640 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:49.286656 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:02:49.286715 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:49.286718 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:49.286773 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:49.286860 (+ 87us) server_negotiation.cc:227] Negotiated authn=JWT
0131 03:02:49.287202 (+ 342us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.287737 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:49.288435 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.288582 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:49.288629 (+ 47us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0131 03:02:49.289336 (+ 707us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0131 03:02:49.289485 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49288: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
W20260131 03:02:49.289680 12072 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49288: JWT verification failed: token expired
W20260131 03:02:49.289672 12071 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:49.286199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:34809 (local address 127.0.0.1:49288)
0131 03:02:49.286496 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:49.286514 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:49.286621 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:49.286897 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:49.286899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:49.286905 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0131 03:02:49.287095 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:49.287101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.287837 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:49.287840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:49.288247 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:49.288253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.288339 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:49.288358 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0131 03:02:49.289450 (+ 1092us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0131 03:02:49.289558 (+ 108us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.22.126:34809: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
W20260131 03:02:49.292171 12072 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:49.291396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.6.22.126:34809)
0131 03:02:49.291530 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:49.291534 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:49.291630 (+ 96us) server_negotiation.cc:408] Connection header received
0131 03:02:49.291766 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:49.291769 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:49.291818 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:49.291898 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:49.292082 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49304: BlockingRecv error: recv got EOF from 127.0.0.1:49304 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260131 03:02:49.295634 12082 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.6.22.126:34809
W20260131 03:02:49.296240 12072 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:49.295349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.6.22.126:34809)
0131 03:02:49.295471 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:49.295475 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:49.295595 (+ 120us) server_negotiation.cc:408] Connection header received
0131 03:02:49.295816 (+ 221us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:49.295820 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:49.295868 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:49.295935 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:49.296146 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49320: BlockingRecv error: recv got EOF from 127.0.0.1:49320 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260131 03:02:49.296382 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 11963
2026-01-31T03:02:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15291 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260131 03:02:49.337036 6233 mini_oidc.cc:150] Starting JWKS server
I20260131 03:02:49.337956 6233 webserver.cc:492] Webserver started at https://127.0.0.1:35559/ using document root <none> and password file <none>
I20260131 03:02:49.338068 6233 mini_oidc.cc:186] Starting OIDC Discovery server
I20260131 03:02:49.338300 6233 webserver.cc:492] Webserver started at http://127.0.0.1:35375/ using document root <none> and password file <none>
2026-01-31T03:02:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:49Z Disabled control of system clock
I20260131 03:02:49.346971 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:35343
--webserver_interface=127.6.22.126
--webserver_port=0
--builtin_ntp_servers=127.6.22.84:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:35343
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35559/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/testchainca.cert with env {}
W20260131 03:02:49.449630 12088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:49.449846 12088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:49.449891 12088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:49.452853 12088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:49.452900 12088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:49.452916 12088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:49.452932 12088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:49.452947 12088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260131 03:02:49.452963 12088 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260131 03:02:49.452976 12088 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35559/jwks.json
I20260131 03:02:49.456977 12088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:35343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:35343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35559/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:49.457899 12088 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:49.458653 12088 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:49.463860 12094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:49.463856 12093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:49.463856 12096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:49.463922 12088 server_base.cc:1060] running on GCE node
I20260131 03:02:49.464504 12088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:49.465404 12088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:49.466550 12088 hybrid_clock.cc:648] HybridClock initialized: now 1769828569466523 us; error 40 us; skew 500 ppm
I20260131 03:02:49.468125 12088 webserver.cc:492] Webserver started at http://127.6.22.126:42347/ using document root <none> and password file <none>
I20260131 03:02:49.468690 12088 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:49.468752 12088 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:49.468943 12088 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:49.470504 12088 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "fee173b4b65640ca818a587883f1f41d"
format_stamp: "Formatted at 2026-01-31 03:02:49 on dist-test-slave-j368"
server_key: "5374eeadad405c365c5890275728e8be"
server_key_iv: "224a691f2bffe3e3cfd3bbd9de9cbdb1"
server_key_version: "encryptionkey@0"
I20260131 03:02:49.470959 12088 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "fee173b4b65640ca818a587883f1f41d"
format_stamp: "Formatted at 2026-01-31 03:02:49 on dist-test-slave-j368"
server_key: "5374eeadad405c365c5890275728e8be"
server_key_iv: "224a691f2bffe3e3cfd3bbd9de9cbdb1"
server_key_version: "encryptionkey@0"
I20260131 03:02:49.474015 12088 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260131 03:02:49.476035 12102 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:49.477001 12088 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260131 03:02:49.477125 12088 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "fee173b4b65640ca818a587883f1f41d"
format_stamp: "Formatted at 2026-01-31 03:02:49 on dist-test-slave-j368"
server_key: "5374eeadad405c365c5890275728e8be"
server_key_iv: "224a691f2bffe3e3cfd3bbd9de9cbdb1"
server_key_version: "encryptionkey@0"
I20260131 03:02:49.477223 12088 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:49.491865 12088 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:49.492441 12088 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:49.492604 12088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:49.499061 12088 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:35343
I20260131 03:02:49.499070 12154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:35343 every 8 connection(s)
I20260131 03:02:49.499935 12088 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:49.501825 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 12088
I20260131 03:02:49.501948 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:49.502180 6233 external_mini_cluster.cc:1468] Setting key 795ec487876a761c7672ba0d7d02c294
I20260131 03:02:49.502625 12155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:49.509137 12155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d: Bootstrap starting.
I20260131 03:02:49.510028 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:49.503239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.6.22.126:35343)
0131 03:02:49.503788 (+ 549us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:49.503797 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:49.503833 (+ 36us) server_negotiation.cc:408] Connection header received
0131 03:02:49.504620 (+ 787us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:49.504642 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:49.505077 (+ 435us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:49.505492 (+ 415us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:49.505975 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.507056 (+ 1081us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:49.507745 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.508013 (+ 268us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:49.508372 (+ 359us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:49.508412 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:49.508423 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:49.508891 (+ 468us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:49.508938 (+ 47us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:49.508964 (+ 26us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:49.509064 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:49.509164 (+ 100us) server_negotiation.cc:300] Negotiation successful
0131 03:02:49.509364 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
I20260131 03:02:49.511574 12155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:49.512320 12155 log.cc:826] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:49.512567 6233 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:02:49.514052 12155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d: No bootstrap required, opened a new log
I20260131 03:02:49.516364 12155 raft_consensus.cc:359] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee173b4b65640ca818a587883f1f41d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 35343 } }
I20260131 03:02:49.516513 12155 raft_consensus.cc:385] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:49.516592 12155 raft_consensus.cc:740] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee173b4b65640ca818a587883f1f41d, State: Initialized, Role: FOLLOWER
I20260131 03:02:49.517009 12155 consensus_queue.cc:260] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee173b4b65640ca818a587883f1f41d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 35343 } }
I20260131 03:02:49.517138 12155 raft_consensus.cc:399] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:49.517197 12155 raft_consensus.cc:493] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:49.517297 12155 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:49.518038 12155 raft_consensus.cc:515] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee173b4b65640ca818a587883f1f41d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 35343 } }
I20260131 03:02:49.518343 12155 leader_election.cc:304] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fee173b4b65640ca818a587883f1f41d; no voters:
I20260131 03:02:49.518589 12155 leader_election.cc:290] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:49.518695 12160 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:49.518899 12160 raft_consensus.cc:697] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [term 1 LEADER]: Becoming Leader. State: Replica: fee173b4b65640ca818a587883f1f41d, State: Running, Role: LEADER
I20260131 03:02:49.519198 12160 consensus_queue.cc:237] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee173b4b65640ca818a587883f1f41d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 35343 } }
I20260131 03:02:49.519572 12155 sys_catalog.cc:565] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:49.520526 12162 sys_catalog.cc:455] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fee173b4b65640ca818a587883f1f41d. Latest consensus state: current_term: 1 leader_uuid: "fee173b4b65640ca818a587883f1f41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee173b4b65640ca818a587883f1f41d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 35343 } } }
I20260131 03:02:49.520637 12162 sys_catalog.cc:458] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:49.521011 12169 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:49.521049 12161 sys_catalog.cc:455] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fee173b4b65640ca818a587883f1f41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee173b4b65640ca818a587883f1f41d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 35343 } } }
I20260131 03:02:49.521138 12161 sys_catalog.cc:458] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:49.523600 12169 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:49.528437 12169 catalog_manager.cc:1357] Generated new cluster ID: a41a4fbc044a41d8b992c5b6302c4ade
I20260131 03:02:49.528503 12169 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:49.549528 12169 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:49.550314 12169 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:49.559253 12169 catalog_manager.cc:6027] T 00000000000000000000000000000000 P fee173b4b65640ca818a587883f1f41d: Generated new TSK 0
I20260131 03:02:49.559849 12169 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260131 03:02:49.583192 12185 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260131 03:02:49.583271 12185 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260131 03:02:49.583798 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:49.569335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.6.22.126:35343)
0131 03:02:49.569448 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:49.569452 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:49.569552 (+ 100us) server_negotiation.cc:408] Connection header received
0131 03:02:49.569678 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:49.569682 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:49.569735 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:49.569821 (+ 86us) server_negotiation.cc:227] Negotiated authn=JWT
0131 03:02:49.570121 (+ 300us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.570863 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:49.571427 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.571610 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:49.571698 (+ 88us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0131 03:02:49.583469 (+ 11771us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0131 03:02:49.583681 (+ 212us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42362: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260131 03:02:49.583865 12158 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42362: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260131 03:02:49.583838 12184 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:49.569242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:35343 (local address 127.0.0.1:42362)
0131 03:02:49.569473 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:49.569484 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:49.569566 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:49.569833 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:49.569835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:49.569841 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0131 03:02:49.570028 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:49.570034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.570969 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:49.570971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:49.571328 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:49.571334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.571409 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:49.571423 (+ 14us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0131 03:02:49.583614 (+ 12191us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0131 03:02:49.583714 (+ 100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.22.126:35343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260131 03:02:49.584444 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 12088
2026-01-31T03:02:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (286 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260131 03:02:49.623118 6233 mini_oidc.cc:150] Starting JWKS server
I20260131 03:02:49.623878 6233 webserver.cc:492] Webserver started at https://127.0.0.1:42641/ using document root <none> and password file <none>
I20260131 03:02:49.623970 6233 mini_oidc.cc:186] Starting OIDC Discovery server
I20260131 03:02:49.624233 6233 webserver.cc:492] Webserver started at http://127.0.0.1:46079/ using document root <none> and password file <none>
2026-01-31T03:02:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:49Z Disabled control of system clock
I20260131 03:02:49.631405 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:38171
--webserver_interface=127.6.22.126
--webserver_port=0
--builtin_ntp_servers=127.6.22.84:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:38171
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42641/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260131 03:02:49.720985 12191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:49.721181 12191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:49.721252 12191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:49.724282 12191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:49.724370 12191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:49.724411 12191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:49.724467 12191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:49.724509 12191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260131 03:02:49.724548 12191 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260131 03:02:49.724581 12191 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42641/jwks.json
I20260131 03:02:49.727833 12191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:38171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:38171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42641/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:49.728596 12191 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:49.729512 12191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:02:49.734123 12191 server_base.cc:1060] running on GCE node
W20260131 03:02:49.734083 12197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:49.734074 12199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:49.734084 12196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:49.734725 12191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:49.735508 12191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:49.736689 12191 hybrid_clock.cc:648] HybridClock initialized: now 1769828569736681 us; error 26 us; skew 500 ppm
I20260131 03:02:49.738143 12191 webserver.cc:492] Webserver started at http://127.6.22.126:39203/ using document root <none> and password file <none>
I20260131 03:02:49.738579 12191 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:49.738644 12191 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:49.738809 12191 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:49.740087 12191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "41fbd92e1c6b4d509eed178c9b8bd6c0"
format_stamp: "Formatted at 2026-01-31 03:02:49 on dist-test-slave-j368"
server_key: "a2f906f911d16ed1a1ddb92066eeffb8"
server_key_iv: "bbd0cbe77d231dca40c1f9f7b1556620"
server_key_version: "encryptionkey@0"
I20260131 03:02:49.740473 12191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "41fbd92e1c6b4d509eed178c9b8bd6c0"
format_stamp: "Formatted at 2026-01-31 03:02:49 on dist-test-slave-j368"
server_key: "a2f906f911d16ed1a1ddb92066eeffb8"
server_key_iv: "bbd0cbe77d231dca40c1f9f7b1556620"
server_key_version: "encryptionkey@0"
I20260131 03:02:49.743072 12191 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:49.744865 12205 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:49.745707 12191 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:49.745811 12191 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "41fbd92e1c6b4d509eed178c9b8bd6c0"
format_stamp: "Formatted at 2026-01-31 03:02:49 on dist-test-slave-j368"
server_key: "a2f906f911d16ed1a1ddb92066eeffb8"
server_key_iv: "bbd0cbe77d231dca40c1f9f7b1556620"
server_key_version: "encryptionkey@0"
I20260131 03:02:49.745888 12191 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:49.760118 12191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:49.760532 12191 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:49.760687 12191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:49.766220 12191 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:38171
I20260131 03:02:49.766228 12257 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:38171 every 8 connection(s)
I20260131 03:02:49.766960 12191 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:49.769209 12258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:49.773622 12258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0: Bootstrap starting.
I20260131 03:02:49.775628 12258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:49.775980 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 12191
I20260131 03:02:49.776083 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:49.776275 6233 external_mini_cluster.cc:1468] Setting key 88d32cd33bfb44fb8bf7930a4cc4d592
I20260131 03:02:49.776268 12258 log.cc:826] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:49.777774 12258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0: No bootstrap required, opened a new log
I20260131 03:02:49.780015 12258 raft_consensus.cc:359] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fbd92e1c6b4d509eed178c9b8bd6c0" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38171 } }
I20260131 03:02:49.780217 12258 raft_consensus.cc:385] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:49.780274 12258 raft_consensus.cc:740] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41fbd92e1c6b4d509eed178c9b8bd6c0, State: Initialized, Role: FOLLOWER
I20260131 03:02:49.780637 12258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fbd92e1c6b4d509eed178c9b8bd6c0" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38171 } }
I20260131 03:02:49.780807 12258 raft_consensus.cc:399] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:49.780866 12258 raft_consensus.cc:493] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:49.780956 12258 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:49.781777 12258 raft_consensus.cc:515] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fbd92e1c6b4d509eed178c9b8bd6c0" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38171 } }
I20260131 03:02:49.782051 12258 leader_election.cc:304] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41fbd92e1c6b4d509eed178c9b8bd6c0; no voters:
I20260131 03:02:49.782270 12258 leader_election.cc:290] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:49.782354 12263 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:49.782527 12263 raft_consensus.cc:697] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [term 1 LEADER]: Becoming Leader. State: Replica: 41fbd92e1c6b4d509eed178c9b8bd6c0, State: Running, Role: LEADER
I20260131 03:02:49.782817 12263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fbd92e1c6b4d509eed178c9b8bd6c0" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38171 } }
I20260131 03:02:49.783136 12258 sys_catalog.cc:565] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:49.783742 12263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41fbd92e1c6b4d509eed178c9b8bd6c0. Latest consensus state: current_term: 1 leader_uuid: "41fbd92e1c6b4d509eed178c9b8bd6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fbd92e1c6b4d509eed178c9b8bd6c0" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38171 } } }
I20260131 03:02:49.783852 12263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:49.784171 12264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41fbd92e1c6b4d509eed178c9b8bd6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fbd92e1c6b4d509eed178c9b8bd6c0" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38171 } } }
I20260131 03:02:49.784314 12264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:49.786145 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:49.777096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.6.22.126:38171)
0131 03:02:49.777571 (+ 475us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:49.777578 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:49.780462 (+ 2884us) server_negotiation.cc:408] Connection header received
0131 03:02:49.780701 (+ 239us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:49.780717 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:49.781009 (+ 292us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:49.781233 (+ 224us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:49.781684 (+ 451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.782452 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:49.783427 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.783876 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:49.784036 (+ 160us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:49.784050 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:49.784061 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:49.784980 (+ 919us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:49.785026 (+ 46us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:49.785044 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:49.785145 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:49.785447 (+ 302us) server_negotiation.cc:300] Negotiation successful
0131 03:02:49.785692 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":117,"threads_started":1}
I20260131 03:02:49.788039 6233 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
W20260131 03:02:49.788465 12278 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:02:49.788528 12278 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:02:49.788580 12269 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:49.789203 12269 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:49.793000 12269 catalog_manager.cc:1357] Generated new cluster ID: 7316caaba3e948b58b68c93652736d93
I20260131 03:02:49.793056 12269 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:49.807304 12269 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:49.808178 12269 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:49.816629 12269 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 41fbd92e1c6b4d509eed178c9b8bd6c0: Generated new TSK 0
I20260131 03:02:49.817265 12269 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260131 03:02:49.846802 12288 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260131 03:02:49.846854 12288 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260131 03:02:49.847666 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:49.826693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.6.22.126:38171)
0131 03:02:49.826808 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:49.826815 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:49.826944 (+ 129us) server_negotiation.cc:408] Connection header received
0131 03:02:49.827071 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:49.827073 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:49.827121 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:49.827195 (+ 74us) server_negotiation.cc:227] Negotiated authn=JWT
0131 03:02:49.827544 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.827940 (+ 396us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:49.828719 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.828836 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:49.828898 (+ 62us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0131 03:02:49.847415 (+ 18517us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0131 03:02:49.847590 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52792: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":44}
W20260131 03:02:49.847724 12262 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52792: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260131 03:02:49.847791 12287 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:49.826637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38171 (local address 127.0.0.1:52792)
0131 03:02:49.826870 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:49.826883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:49.826975 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:49.827218 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:49.827222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:49.827229 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0131 03:02:49.827455 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:49.827462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.828044 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:49.828048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:49.828589 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:49.828597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:49.828723 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:49.828743 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0131 03:02:49.847547 (+ 18804us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0131 03:02:49.847648 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.22.126:38171: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260131 03:02:49.848054 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 12191
2026-01-31T03:02:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (263 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-01-31T03:02:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:49Z Disabled control of system clock
I20260131 03:02:49.892290 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:33931
--webserver_interface=127.6.22.126
--webserver_port=0
--builtin_ntp_servers=127.6.22.84:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:33931
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/testchainca.cert with env {}
W20260131 03:02:49.980995 12292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:49.981151 12292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:49.981230 12292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:49.983981 12292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:49.984023 12292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:49.984038 12292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:49.984054 12292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:49.984105 12292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260131 03:02:49.984144 12292 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260131 03:02:49.984191 12292 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260131 03:02:49.987704 12292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:33931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:33931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:49.988530 12292 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:49.989214 12292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:49.993866 12300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:49.993877 12297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:49.993877 12298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:49.994056 12292 server_base.cc:1060] running on GCE node
I20260131 03:02:49.994560 12292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:49.995414 12292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:49.996551 12292 hybrid_clock.cc:648] HybridClock initialized: now 1769828569996543 us; error 22 us; skew 500 ppm
I20260131 03:02:49.998090 12292 webserver.cc:492] Webserver started at http://127.6.22.126:44947/ using document root <none> and password file <none>
I20260131 03:02:49.998551 12292 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:49.998607 12292 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:49.998796 12292 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:50.000226 12292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "5cd33f03e73b44879bd7d43f306541d6"
format_stamp: "Formatted at 2026-01-31 03:02:49 on dist-test-slave-j368"
server_key: "4efde6085767ab0cfa6497562535bb5e"
server_key_iv: "4d6bd59f6d2f7a65398f958987289d35"
server_key_version: "encryptionkey@0"
I20260131 03:02:50.000694 12292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "5cd33f03e73b44879bd7d43f306541d6"
format_stamp: "Formatted at 2026-01-31 03:02:49 on dist-test-slave-j368"
server_key: "4efde6085767ab0cfa6497562535bb5e"
server_key_iv: "4d6bd59f6d2f7a65398f958987289d35"
server_key_version: "encryptionkey@0"
I20260131 03:02:50.003434 12292 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:50.005338 12306 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:50.006255 12292 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:50.006382 12292 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "5cd33f03e73b44879bd7d43f306541d6"
format_stamp: "Formatted at 2026-01-31 03:02:49 on dist-test-slave-j368"
server_key: "4efde6085767ab0cfa6497562535bb5e"
server_key_iv: "4d6bd59f6d2f7a65398f958987289d35"
server_key_version: "encryptionkey@0"
I20260131 03:02:50.006472 12292 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:50.027928 12292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:50.028647 12292 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:50.028833 12292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:50.036810 12292 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:33931
I20260131 03:02:50.036813 12358 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:33931 every 8 connection(s)
I20260131 03:02:50.037968 12292 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:50.040395 12359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:50.045059 12359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6: Bootstrap starting.
I20260131 03:02:50.046176 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 12292
I20260131 03:02:50.046288 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:50.046489 6233 external_mini_cluster.cc:1468] Setting key 64d7cc227d4d8126d04ebd7c0f1f9174
I20260131 03:02:50.047405 12359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:50.048053 12359 log.cc:826] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:50.049645 12359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6: No bootstrap required, opened a new log
I20260131 03:02:50.051844 12359 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd33f03e73b44879bd7d43f306541d6" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 33931 } }
I20260131 03:02:50.052024 12359 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:50.052088 12359 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cd33f03e73b44879bd7d43f306541d6, State: Initialized, Role: FOLLOWER
I20260131 03:02:50.052464 12359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd33f03e73b44879bd7d43f306541d6" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 33931 } }
I20260131 03:02:50.052592 12359 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:50.052681 12359 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:50.052771 12359 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:50.052825 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:50.047320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.6.22.126:33931)
0131 03:02:50.047829 (+ 509us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:50.047837 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:50.047866 (+ 29us) server_negotiation.cc:408] Connection header received
0131 03:02:50.048373 (+ 507us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:50.048395 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:50.048728 (+ 333us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:50.049033 (+ 305us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:50.049436 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:50.050123 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:50.050702 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:50.050953 (+ 251us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:50.051350 (+ 397us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:50.051371 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:50.051379 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:02:50.051690 (+ 311us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:50.051742 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:02:50.051755 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:50.051856 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:50.051991 (+ 135us) server_negotiation.cc:300] Negotiation successful
0131 03:02:50.052212 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":100,"threads_started":1}
I20260131 03:02:50.053653 12359 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd33f03e73b44879bd7d43f306541d6" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 33931 } }
I20260131 03:02:50.053958 12359 leader_election.cc:304] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cd33f03e73b44879bd7d43f306541d6; no voters:
I20260131 03:02:50.054270 12359 leader_election.cc:290] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:50.054391 12364 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:50.054592 12364 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [term 1 LEADER]: Becoming Leader. State: Replica: 5cd33f03e73b44879bd7d43f306541d6, State: Running, Role: LEADER
I20260131 03:02:50.054888 12364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd33f03e73b44879bd7d43f306541d6" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 33931 } }
I20260131 03:02:50.055239 6233 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:02:50.055349 12359 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:50.056241 12365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cd33f03e73b44879bd7d43f306541d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd33f03e73b44879bd7d43f306541d6" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 33931 } } }
I20260131 03:02:50.056366 12365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:50.056558 12365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cd33f03e73b44879bd7d43f306541d6. Latest consensus state: current_term: 1 leader_uuid: "5cd33f03e73b44879bd7d43f306541d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd33f03e73b44879bd7d43f306541d6" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 33931 } } }
I20260131 03:02:50.056643 12365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6 [sys.catalog]: This master's current role is: LEADER
W20260131 03:02:50.059592 12379 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:02:50.059670 12379 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:02:50.059739 12373 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:50.060370 12373 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:50.064379 12373 catalog_manager.cc:1357] Generated new cluster ID: cfa6a8fa8b2643559f429cbfd4d98b16
I20260131 03:02:50.064442 12373 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:50.073146 12373 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:50.073900 12373 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:50.084506 12373 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5cd33f03e73b44879bd7d43f306541d6: Generated new TSK 0
I20260131 03:02:50.085206 12373 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:02:50.132014 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:50.099732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.6.22.126:33931)
0131 03:02:50.099897 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:50.099901 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:50.100137 (+ 236us) server_negotiation.cc:408] Connection header received
0131 03:02:50.100402 (+ 265us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:50.100406 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:50.100459 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:50.100543 (+ 84us) server_negotiation.cc:227] Negotiated authn=JWT
0131 03:02:50.101074 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:50.101777 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:50.102542 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:50.102726 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:50.102813 (+ 87us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0131 03:02:50.131575 (+ 28762us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0131 03:02:50.131858 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40096: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":77}
W20260131 03:02:50.132050 12388 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:50.099593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:33931 (local address 127.0.0.1:40096)
0131 03:02:50.100061 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:50.100076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:50.100210 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:50.100638 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:50.100641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:50.100670 (+ 29us) client_negotiation.cc:189] Negotiated authn=JWT
0131 03:02:50.100901 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:50.100909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:50.101906 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:50.101910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:50.102442 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:50.102452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:50.102541 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:50.102557 (+ 16us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0131 03:02:50.131772 (+ 29215us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0131 03:02:50.131904 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.22.126:33931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":119,"threads_started":1}
W20260131 03:02:50.132094 12362 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40096: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
I20260131 03:02:50.132596 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 12292
2026-01-31T03:02:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (263 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (104 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (106 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:02:50 dist-test-slave-j368 krb5kdc[12400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:02:50 dist-test-slave-j368 krb5kdc[12400](info): set up 2 sockets
Jan 31 03:02:50 dist-test-slave-j368 krb5kdc[12400](info): commencing operation
krb5kdc: starting...
W20260131 03:02:52.194049 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.832s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:02:52 dist-test-slave-j368 krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:02:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:52.303200 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:37395
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:37395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:52.401190 12416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:52.401432 12416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:52.401489 12416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:52.404721 12416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:52.404781 12416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:52.404804 12416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:52.404824 12416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:52.404840 12416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:52.409137 12416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:37395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:37395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:52.410187 12416 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:52.410975 12416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:52.416296 12422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:52.416431 12416 server_base.cc:1060] running on GCE node
W20260131 03:02:52.416316 12421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:52.416314 12424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:52.417111 12416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:52.417969 12416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:52.419143 12416 hybrid_clock.cc:648] HybridClock initialized: now 1769828572419131 us; error 27 us; skew 500 ppm
Jan 31 03:02:52 dist-test-slave-j368 krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:52.421960 12416 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:02:52.423450 12416 webserver.cc:492] Webserver started at http://127.6.22.126:38835/ using document root <none> and password file <none>
I20260131 03:02:52.424160 12416 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:52.424247 12416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:52.424489 12416 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:52.426612 12416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "e1b6aa061bd0482cb08b721ed0247337"
format_stamp: "Formatted at 2026-01-31 03:02:52 on dist-test-slave-j368"
server_key: "e78190b8de935b398a5c2093e3570c7e"
server_key_iv: "ba31ce86b5e3d67b22dfcf3fe93324c4"
server_key_version: "encryptionkey@0"
I20260131 03:02:52.427234 12416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "e1b6aa061bd0482cb08b721ed0247337"
format_stamp: "Formatted at 2026-01-31 03:02:52 on dist-test-slave-j368"
server_key: "e78190b8de935b398a5c2093e3570c7e"
server_key_iv: "ba31ce86b5e3d67b22dfcf3fe93324c4"
server_key_version: "encryptionkey@0"
I20260131 03:02:52.431434 12416 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260131 03:02:52.434001 12431 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:52.434988 12416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:02:52.435119 12416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "e1b6aa061bd0482cb08b721ed0247337"
format_stamp: "Formatted at 2026-01-31 03:02:52 on dist-test-slave-j368"
server_key: "e78190b8de935b398a5c2093e3570c7e"
server_key_iv: "ba31ce86b5e3d67b22dfcf3fe93324c4"
server_key_version: "encryptionkey@0"
I20260131 03:02:52.435243 12416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:52.453197 12416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:52.455783 12416 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:52.455968 12416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:52.463214 12416 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:37395
I20260131 03:02:52.463220 12483 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:37395 every 8 connection(s)
I20260131 03:02:52.464182 12416 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:52.466862 12484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:52.467826 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 12416
I20260131 03:02:52.467965 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:52.468216 6233 external_mini_cluster.cc:1468] Setting key cdabba92f4b97113a0760ab9c97d2654
I20260131 03:02:52.473148 12484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337: Bootstrap starting.
I20260131 03:02:52.475332 12484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337: Neither blocks nor log segments found. Creating new log.
Jan 31 03:02:52 dist-test-slave-j368 krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:52.476078 12484 log.cc:826] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:52.477957 12484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337: No bootstrap required, opened a new log
I20260131 03:02:52.480170 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:52.469473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41162 (local address 127.6.22.126:37395)
0131 03:02:52.469888 (+ 415us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:52.469901 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:52.469942 (+ 41us) server_negotiation.cc:408] Connection header received
0131 03:02:52.470593 (+ 651us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:52.470615 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:52.470970 (+ 355us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:52.471292 (+ 322us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:52.472088 (+ 796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:52.473265 (+ 1177us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:52.473883 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:52.474163 (+ 280us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:52.476090 (+ 1927us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:52.476111 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:52.476122 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:52.476163 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:52.477850 (+ 1687us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:52.478340 (+ 490us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:52.478345 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:52.478349 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:52.478415 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:52.478646 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:52.478649 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:52.478651 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:52.478949 (+ 298us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:52.479158 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:52.479418 (+ 260us) server_negotiation.cc:300] Negotiation successful
0131 03:02:52.479642 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
I20260131 03:02:52.480433 12484 raft_consensus.cc:359] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b6aa061bd0482cb08b721ed0247337" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 37395 } }
I20260131 03:02:52.480638 12484 raft_consensus.cc:385] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:52.480718 12484 raft_consensus.cc:740] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b6aa061bd0482cb08b721ed0247337, State: Initialized, Role: FOLLOWER
I20260131 03:02:52.481124 12484 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b6aa061bd0482cb08b721ed0247337" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 37395 } }
I20260131 03:02:52.481276 12484 raft_consensus.cc:399] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:52.481338 12484 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:52.481449 12484 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:52.482657 12484 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b6aa061bd0482cb08b721ed0247337" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 37395 } }
I20260131 03:02:52.482985 12484 leader_election.cc:304] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1b6aa061bd0482cb08b721ed0247337; no voters:
I20260131 03:02:52.483234 12484 leader_election.cc:290] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:52.483335 12489 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:52.483546 12489 raft_consensus.cc:697] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [term 1 LEADER]: Becoming Leader. State: Replica: e1b6aa061bd0482cb08b721ed0247337, State: Running, Role: LEADER
I20260131 03:02:52.483840 12489 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b6aa061bd0482cb08b721ed0247337" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 37395 } }
I20260131 03:02:52.484287 12484 sys_catalog.cc:565] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:52.485374 12491 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1b6aa061bd0482cb08b721ed0247337. Latest consensus state: current_term: 1 leader_uuid: "e1b6aa061bd0482cb08b721ed0247337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b6aa061bd0482cb08b721ed0247337" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 37395 } } }
I20260131 03:02:52.485484 12491 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:52.485818 12490 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1b6aa061bd0482cb08b721ed0247337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b6aa061bd0482cb08b721ed0247337" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 37395 } } }
I20260131 03:02:52.485898 12498 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:52.485956 12490 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:52.488914 12498 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:52.493722 12498 catalog_manager.cc:1357] Generated new cluster ID: 3e4685d70abb401abbd733f2c698b3da
I20260131 03:02:52.493779 12498 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:52.506325 12498 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:52.506994 12498 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:52.512997 12498 catalog_manager.cc:6027] T 00000000000000000000000000000000 P e1b6aa061bd0482cb08b721ed0247337: Generated new TSK 0
I20260131 03:02:52.513504 12498 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:52.559116 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:37395
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:52.650622 12512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:52.650776 12512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:52.650822 12512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:52.653740 12512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:52.653785 12512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:52.653885 12512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:02:52.657542 12512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:37395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:52.658382 12512 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:52.659116 12512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:52.664486 12518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:52.664455 12517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:52.664619 12512 server_base.cc:1060] running on GCE node
W20260131 03:02:52.664449 12520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:52.665019 12512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:52.665488 12512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:52.666656 12512 hybrid_clock.cc:648] HybridClock initialized: now 1769828572666644 us; error 32 us; skew 500 ppm
Jan 31 03:02:52 dist-test-slave-j368 krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:52.668690 12512 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:02:52.669516 12512 webserver.cc:492] Webserver started at http://127.6.22.65:34223/ using document root <none> and password file <none>
I20260131 03:02:52.669984 12512 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:52.670045 12512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:52.670221 12512 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:52.671618 12512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "14a60d9ae39c4a63b5118265047ddf7d"
format_stamp: "Formatted at 2026-01-31 03:02:52 on dist-test-slave-j368"
server_key: "1f7a1e1b3666f03494513152250baed7"
server_key_iv: "b53cb480f7e6d328ab5ee7ee7134691c"
server_key_version: "encryptionkey@0"
I20260131 03:02:52.672021 12512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "14a60d9ae39c4a63b5118265047ddf7d"
format_stamp: "Formatted at 2026-01-31 03:02:52 on dist-test-slave-j368"
server_key: "1f7a1e1b3666f03494513152250baed7"
server_key_iv: "b53cb480f7e6d328ab5ee7ee7134691c"
server_key_version: "encryptionkey@0"
I20260131 03:02:52.674826 12512 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260131 03:02:52.676630 12527 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:52.677515 12512 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:52.677628 12512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "14a60d9ae39c4a63b5118265047ddf7d"
format_stamp: "Formatted at 2026-01-31 03:02:52 on dist-test-slave-j368"
server_key: "1f7a1e1b3666f03494513152250baed7"
server_key_iv: "b53cb480f7e6d328ab5ee7ee7134691c"
server_key_version: "encryptionkey@0"
I20260131 03:02:52.677716 12512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:52.690366 12512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:52.692327 12512 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:52.692471 12512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:52.692925 12512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:52.693606 12512 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:52.693663 12512 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:52.693713 12512 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:52.693747 12512 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:52.701123 12512 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:44173
I20260131 03:02:52.701169 12640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:44173 every 8 connection(s)
I20260131 03:02:52.701885 12512 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:02:52.703238 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 12512
I20260131 03:02:52.703357 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:02:52.703562 6233 external_mini_cluster.cc:1468] Setting key 355034311c4cda1ebe7b1b780f2184fd
Jan 31 03:02:52 dist-test-slave-j368 krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
I20260131 03:02:52.712882 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:52.703319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:59653 (local address 127.6.22.126:37395)
0131 03:02:52.703438 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:52.703443 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:52.703971 (+ 528us) server_negotiation.cc:408] Connection header received
0131 03:02:52.704799 (+ 828us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:52.704803 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:52.704855 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:52.704950 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:52.706343 (+ 1393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:52.706787 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:52.707576 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:52.707774 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:52.709915 (+ 2141us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:52.709933 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:52.709938 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:52.709969 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:52.711330 (+ 1361us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:52.711711 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:52.711714 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:52.711716 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:52.711760 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:52.712027 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:52.712031 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:52.712033 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:52.712221 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:52.712299 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:52.712688 (+ 389us) server_negotiation.cc:300] Negotiation successful
0131 03:02:52.712782 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260131 03:02:52.713387 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:52.703513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:37395 (local address 127.6.22.65:59653)
0131 03:02:52.703864 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:52.703897 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:52.704573 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:52.705085 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:52.705092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:52.705410 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:52.706210 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:52.706222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:52.706889 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:52.706894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:52.707451 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:52.707462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:52.707962 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:52.708418 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:52.708449 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:52.709790 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:52.711431 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:52.711434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:52.711440 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:52.711626 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:52.711844 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:52.711846 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:52.711847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:52.711952 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:52.712311 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:52.712315 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:52.712483 (+ 168us) client_negotiation.cc:769] Sending connection context
0131 03:02:52.712618 (+ 135us) client_negotiation.cc:240] Negotiation successful
0131 03:02:52.712825 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
I20260131 03:02:52.714435 12641 heartbeater.cc:344] Connected to a master server at 127.6.22.126:37395
I20260131 03:02:52.714658 12641 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:52.715085 12641 heartbeater.cc:507] Master 127.6.22.126:37395 requested a full tablet report, sending...
I20260131 03:02:52.716323 12448 ts_manager.cc:194] Registered new tserver with Master: 14a60d9ae39c4a63b5118265047ddf7d (127.6.22.65:44173)
I20260131 03:02:52.717396 12448 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:59653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:52.742440 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:37395
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:52.834100 12648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:52.834247 12648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:52.834321 12648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:52.837409 12648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:52.837456 12648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:52.837567 12648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:02:52.841126 12648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:37395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:52.841964 12648 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:52.842676 12648 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:52.848227 12653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:52.848208 12656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:52.848212 12654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:52.848474 12648 server_base.cc:1060] running on GCE node
I20260131 03:02:52.848809 12648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:52.849242 12648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:52.850373 12648 hybrid_clock.cc:648] HybridClock initialized: now 1769828572850356 us; error 28 us; skew 500 ppm
Jan 31 03:02:52 dist-test-slave-j368 krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:52.852416 12648 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:02:52.853273 12648 webserver.cc:492] Webserver started at http://127.6.22.66:44483/ using document root <none> and password file <none>
I20260131 03:02:52.853726 12648 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:52.853770 12648 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:52.853889 12648 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:52.855304 12648 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "39fd02f3da0b46c68d29754f81b10c64"
format_stamp: "Formatted at 2026-01-31 03:02:52 on dist-test-slave-j368"
server_key: "1b7e5ff90e83a592d18aba237b264596"
server_key_iv: "64cc5a83d22bcd3ea6cb9877c9c1a98c"
server_key_version: "encryptionkey@0"
I20260131 03:02:52.855701 12648 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "39fd02f3da0b46c68d29754f81b10c64"
format_stamp: "Formatted at 2026-01-31 03:02:52 on dist-test-slave-j368"
server_key: "1b7e5ff90e83a592d18aba237b264596"
server_key_iv: "64cc5a83d22bcd3ea6cb9877c9c1a98c"
server_key_version: "encryptionkey@0"
I20260131 03:02:52.858508 12648 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260131 03:02:52.860209 12663 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:52.861088 12648 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:52.861192 12648 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "39fd02f3da0b46c68d29754f81b10c64"
format_stamp: "Formatted at 2026-01-31 03:02:52 on dist-test-slave-j368"
server_key: "1b7e5ff90e83a592d18aba237b264596"
server_key_iv: "64cc5a83d22bcd3ea6cb9877c9c1a98c"
server_key_version: "encryptionkey@0"
I20260131 03:02:52.861269 12648 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:52.869992 12648 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:52.872051 12648 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:52.872212 12648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:52.872687 12648 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:52.873392 12648 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:52.873435 12648 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:52.873488 12648 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:52.873518 12648 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:52.881376 12648 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:45039
I20260131 03:02:52.881456 12776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:45039 every 8 connection(s)
I20260131 03:02:52.882200 12648 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:02:52.886193 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 12648
I20260131 03:02:52.886283 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:02:52.886513 6233 external_mini_cluster.cc:1468] Setting key 315475d324a98fb8fba09009510c6fbc
Jan 31 03:02:52 dist-test-slave-j368 krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:52.893821 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:52.883577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:33027 (local address 127.6.22.126:37395)
0131 03:02:52.883692 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:52.883695 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:52.884202 (+ 507us) server_negotiation.cc:408] Connection header received
0131 03:02:52.885088 (+ 886us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:52.885092 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:52.885148 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:52.885244 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:52.886814 (+ 1570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:52.887310 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:52.888003 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:52.888128 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:52.890563 (+ 2435us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:52.890577 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:52.890579 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:52.890600 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:52.892138 (+ 1538us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:52.892542 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:52.892548 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:52.892551 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:52.892606 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:52.892866 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:52.892869 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:52.892871 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:52.893039 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:52.893141 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:52.893544 (+ 403us) server_negotiation.cc:300] Negotiation successful
0131 03:02:52.893658 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260131 03:02:52.894196 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:52.883771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:37395 (local address 127.6.22.66:33027)
0131 03:02:52.884092 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:52.884118 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:52.884891 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:52.885383 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:52.885391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:52.885697 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:52.886691 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:52.886704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:52.887454 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:52.887457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:52.887897 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:52.887908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:52.888098 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:52.888577 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:52.888614 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:52.890433 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:52.892249 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:52.892253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:52.892259 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:52.892454 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:52.892715 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:52.892717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:52.892718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:52.892784 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:52.893132 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:52.893136 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:52.893321 (+ 185us) client_negotiation.cc:769] Sending connection context
0131 03:02:52.893467 (+ 146us) client_negotiation.cc:240] Negotiation successful
0131 03:02:52.893628 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
I20260131 03:02:52.895390 12777 heartbeater.cc:344] Connected to a master server at 127.6.22.126:37395
I20260131 03:02:52.895593 12777 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:52.896085 12777 heartbeater.cc:507] Master 127.6.22.126:37395 requested a full tablet report, sending...
I20260131 03:02:52.896930 12448 ts_manager.cc:194] Registered new tserver with Master: 39fd02f3da0b46c68d29754f81b10c64 (127.6.22.66:45039)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
I20260131 03:02:52.897576 12448 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:33027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:02:52.925415 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:37395
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:53.015031 12784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:53.015205 12784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:53.015257 12784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:53.018215 12784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:53.018260 12784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:53.018323 12784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:02:53.022269 12784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:37395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:53.023095 12784 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:53.023803 12784 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:02:53.029840 12784 server_base.cc:1060] running on GCE node
W20260131 03:02:53.029791 12792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:53.029800 12789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:53.029841 12790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:53.030395 12784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:53.030903 12784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:53.032043 12784 hybrid_clock.cc:648] HybridClock initialized: now 1769828573032025 us; error 32 us; skew 500 ppm
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828573, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:53.034482 12784 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:02:53.035382 12784 webserver.cc:492] Webserver started at http://127.6.22.67:45545/ using document root <none> and password file <none>
I20260131 03:02:53.035877 12784 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:53.035920 12784 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:53.036106 12784 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:53.037674 12784 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "102b13e40e83403e97b2610faadf555a"
format_stamp: "Formatted at 2026-01-31 03:02:53 on dist-test-slave-j368"
server_key: "5948f722606efd512a259992db293bd3"
server_key_iv: "c5a3eee366e74230f0f759225b3a2f9b"
server_key_version: "encryptionkey@0"
I20260131 03:02:53.038120 12784 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "102b13e40e83403e97b2610faadf555a"
format_stamp: "Formatted at 2026-01-31 03:02:53 on dist-test-slave-j368"
server_key: "5948f722606efd512a259992db293bd3"
server_key_iv: "c5a3eee366e74230f0f759225b3a2f9b"
server_key_version: "encryptionkey@0"
I20260131 03:02:53.041239 12784 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260131 03:02:53.043253 12799 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:53.044162 12784 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260131 03:02:53.044283 12784 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "102b13e40e83403e97b2610faadf555a"
format_stamp: "Formatted at 2026-01-31 03:02:53 on dist-test-slave-j368"
server_key: "5948f722606efd512a259992db293bd3"
server_key_iv: "c5a3eee366e74230f0f759225b3a2f9b"
server_key_version: "encryptionkey@0"
I20260131 03:02:53.044366 12784 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:53.068974 12784 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:53.071616 12784 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:53.071789 12784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:53.072278 12784 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:53.073114 12784 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:53.073179 12784 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:53.073239 12784 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:53.073282 12784 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:53.082461 12784 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:37365
I20260131 03:02:53.082494 12912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:37365 every 8 connection(s)
I20260131 03:02:53.083427 12784 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:02:53.089717 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 12784
I20260131 03:02:53.089855 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:02:53.090090 6233 external_mini_cluster.cc:1468] Setting key 7362dd084a44d77b000fb3b8f10311f9
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828573, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:02:53.095168 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:53.085153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:35611 (local address 127.6.22.126:37395)
0131 03:02:53.085277 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:53.085281 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:53.085971 (+ 690us) server_negotiation.cc:408] Connection header received
0131 03:02:53.086834 (+ 863us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:53.086838 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:53.086900 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:53.087000 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:53.088445 (+ 1445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:53.088922 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:53.089553 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:53.089720 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:53.091829 (+ 2109us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:53.091847 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:53.091853 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:53.091883 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:53.093339 (+ 1456us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:53.093872 (+ 533us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:53.093877 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:53.093879 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:53.093925 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:53.094204 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:53.094208 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:53.094210 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:53.094348 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:53.094465 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:53.094901 (+ 436us) server_negotiation.cc:300] Negotiation successful
0131 03:02:53.095012 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260131 03:02:53.095707 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:53.085437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:37395 (local address 127.6.22.67:35611)
0131 03:02:53.085843 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:53.085876 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:53.086663 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:53.087139 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:53.087146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:53.087482 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:53.088297 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:53.088309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:53.089024 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:53.089031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:53.089400 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:53.089432 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:53.089658 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:53.090228 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:53.090251 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:53.091705 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:53.093481 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:53.093486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:53.093514 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:53.093771 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:53.094013 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:53.094016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:53.094019 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:53.094099 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:53.094471 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:53.094480 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:53.094682 (+ 202us) client_negotiation.cc:769] Sending connection context
0131 03:02:53.094867 (+ 185us) client_negotiation.cc:240] Negotiation successful
0131 03:02:53.095096 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260131 03:02:53.096527 12913 heartbeater.cc:344] Connected to a master server at 127.6.22.126:37395
I20260131 03:02:53.096760 12913 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:53.097219 12913 heartbeater.cc:507] Master 127.6.22.126:37395 requested a full tablet report, sending...
I20260131 03:02:53.098093 12448 ts_manager.cc:194] Registered new tserver with Master: 102b13e40e83403e97b2610faadf555a (127.6.22.67:37365)
I20260131 03:02:53.098598 12448 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:35611
I20260131 03:02:53.103382 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/test-admin.keytab.
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:53.115967 6233 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260131 03:02:53.118405 12920 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:53.117133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.6.22.65:44173)
0131 03:02:53.117448 (+ 315us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:53.117454 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:53.117478 (+ 24us) server_negotiation.cc:408] Connection header received
0131 03:02:53.117526 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:53.117530 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:53.117668 (+ 138us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:53.117784 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:53.118224 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49026: BlockingRecv error: recv got EOF from 127.0.0.1:49026 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":118,"threads_started":1}
W20260131 03:02:53.118479 6233 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/krb5cc' of type FILE
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:53.119472 6233 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:53.120329 6233 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.65@KRBTEST.COM
I20260131 03:02:53.129374 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:53.121294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.6.22.65:44173)
0131 03:02:53.121462 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:53.121491 (+ 29us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:53.121508 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:02:53.121574 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:53.121577 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:53.121638 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:53.121712 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:53.122248 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:53.122840 (+ 592us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:53.123345 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:53.123487 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:53.125347 (+ 1860us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:53.125366 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:53.125376 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:53.125397 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:53.127477 (+ 2080us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:53.127983 (+ 506us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:53.127988 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:53.127989 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:53.128049 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:53.128296 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:53.128299 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:53.128301 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:53.128567 (+ 266us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:53.128856 (+ 289us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:53.129012 (+ 156us) server_negotiation.cc:300] Negotiation successful
0131 03:02:53.129188 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260131 03:02:53.134208 12920 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:53.132807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.6.22.65:44173)
0131 03:02:53.132931 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:53.132955 (+ 24us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:53.133119 (+ 164us) server_negotiation.cc:408] Connection header received
0131 03:02:53.133277 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:53.133281 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:53.133328 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:53.133408 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:53.134068 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49052: BlockingRecv error: recv got EOF from 127.0.0.1:49052 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260131 03:02:53.134358 6233 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/krb5cc' of type FILE
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:53.135430 6233 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:53.136312 6233 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.65@KRBTEST.COM
I20260131 03:02:53.143143 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:53.137254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49066 (local address 127.6.22.65:44173)
0131 03:02:53.137395 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:53.137399 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:53.137416 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:02:53.137547 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:53.137549 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:53.137597 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:53.137665 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:53.138208 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:53.138638 (+ 430us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:53.139126 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:53.139246 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:53.140890 (+ 1644us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:53.140903 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:53.140905 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:53.140924 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:53.142099 (+ 1175us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:53.142410 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:53.142413 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:53.142414 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:53.142449 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:53.142621 (+ 172us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:53.142623 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:53.142624 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:53.142739 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:53.142828 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:53.142965 (+ 137us) server_negotiation.cc:300] Negotiation successful
0131 03:02:53.143041 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260131 03:02:53.145519 12920 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:02:53.144581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.6.22.65:44173)
0131 03:02:53.144735 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:53.144740 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:53.144838 (+ 98us) server_negotiation.cc:408] Connection header received
0131 03:02:53.144965 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:53.144969 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:53.145008 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:53.145091 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:53.145429 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49074: BlockingRecv error: recv got EOF from 127.0.0.1:49074 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260131 03:02:53.145648 6233 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828516572523-6233-0/krb5kdc/krb5cc' of type FILE
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:53.146492 6233 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:53.147217 6233 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.65@KRBTEST.COM
I20260131 03:02:53.154039 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:53.147812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.6.22.65:44173)
0131 03:02:53.147912 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:53.147916 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:53.147992 (+ 76us) server_negotiation.cc:408] Connection header received
0131 03:02:53.148100 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:53.148102 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:53.148131 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:53.148198 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:53.148743 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:53.149135 (+ 392us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:53.149661 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:53.149838 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:53.151478 (+ 1640us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:53.151490 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:53.151492 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:53.151512 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:53.152772 (+ 1260us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:53.153114 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:53.153116 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:53.153118 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:53.153151 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:53.153352 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:53.153353 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:53.153354 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:53.153449 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:53.153528 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:53.153727 (+ 199us) server_negotiation.cc:300] Negotiation successful
0131 03:02:53.153889 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:02:53.154999 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 12512
I20260131 03:02:53.161265 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 12648
I20260131 03:02:53.166178 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 12784
I20260131 03:02:53.170775 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 12416
2026-01-31T03:02:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2830 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12933](info): set up 2 sockets
Jan 31 03:02:53 dist-test-slave-j368 krb5kdc[12933](info): commencing operation
krb5kdc: starting...
W20260131 03:02:54.998183 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.806s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:02:55 dist-test-slave-j368 krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:02:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:02:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
I20260131 03:02:55.126878 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:36467
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:36467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:55.226171 12949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:55.226420 12949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:55.226518 12949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:55.229235 12949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:55.229323 12949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:55.229370 12949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:55.229434 12949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:55.229471 12949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:55.234016 12949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:36467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:36467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:55.235762 12949 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:55.236812 12949 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:55.242115 12954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:55.242106 12955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:55.242103 12957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:55.242453 12949 server_base.cc:1060] running on GCE node
I20260131 03:02:55.242823 12949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:55.243562 12949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:55.244818 12949 hybrid_clock.cc:648] HybridClock initialized: now 1769828575244802 us; error 36 us; skew 500 ppm
Jan 31 03:02:55 dist-test-slave-j368 krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:55.247054 12949 init.cc:377] Logged in from keytab as oryx/127.6.22.126@KRBTEST.COM (short username oryx)
I20260131 03:02:55.247911 12949 webserver.cc:492] Webserver started at http://127.6.22.126:35973/ using document root <none> and password file <none>
I20260131 03:02:55.248373 12949 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:55.248440 12949 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:55.248610 12949 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:55.250228 12949 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "2abf73bb02194843aaeb35f43d12cf52"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "ea8f9f8380a639a8b0f11dd855573842"
server_key_iv: "a2f89c131b2cd7115aee41b7c9e9ca69"
server_key_version: "encryptionkey@0"
I20260131 03:02:55.250725 12949 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "2abf73bb02194843aaeb35f43d12cf52"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "ea8f9f8380a639a8b0f11dd855573842"
server_key_iv: "a2f89c131b2cd7115aee41b7c9e9ca69"
server_key_version: "encryptionkey@0"
I20260131 03:02:55.254096 12949 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260131 03:02:55.256088 12964 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:55.256995 12949 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:55.257143 12949 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "2abf73bb02194843aaeb35f43d12cf52"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "ea8f9f8380a639a8b0f11dd855573842"
server_key_iv: "a2f89c131b2cd7115aee41b7c9e9ca69"
server_key_version: "encryptionkey@0"
I20260131 03:02:55.257246 12949 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:55.285331 12949 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:55.290310 12949 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:55.290526 12949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:55.297670 12949 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:36467
I20260131 03:02:55.297674 13016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:36467 every 8 connection(s)
I20260131 03:02:55.298712 12949 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:02:55.301265 13017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:55.302163 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 12949
I20260131 03:02:55.302299 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:02:55.302557 6233 external_mini_cluster.cc:1468] Setting key c0a5b5a9aa8c13829adb37f27f7d1268
I20260131 03:02:55.307077 13017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52: Bootstrap starting.
I20260131 03:02:55.309648 13017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52: Neither blocks nor log segments found. Creating new log.
Jan 31 03:02:55 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:02:55.310417 13017 log.cc:826] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:55.312511 13017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52: No bootstrap required, opened a new log
I20260131 03:02:55.315258 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:55.303706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.6.22.126:36467)
0131 03:02:55.304070 (+ 364us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:55.304079 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:55.304111 (+ 32us) server_negotiation.cc:408] Connection header received
0131 03:02:55.304606 (+ 495us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:55.304619 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:55.304916 (+ 297us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:55.305164 (+ 248us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:55.306173 (+ 1009us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:55.306787 (+ 614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:55.307486 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:55.307680 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:55.310710 (+ 3030us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:55.310728 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:55.310742 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:55.310788 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:55.312920 (+ 2132us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:55.313442 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:55.313446 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:55.313450 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:55.313519 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:55.313749 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:55.313751 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:55.313752 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:55.314033 (+ 281us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:55.314179 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:55.314446 (+ 267us) server_negotiation.cc:300] Negotiation successful
0131 03:02:55.314684 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":128,"threads_started":1}
I20260131 03:02:55.315435 13017 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } }
I20260131 03:02:55.315655 13017 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:55.315740 13017 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2abf73bb02194843aaeb35f43d12cf52, State: Initialized, Role: FOLLOWER
I20260131 03:02:55.316155 13017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } }
I20260131 03:02:55.316293 13017 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:02:55.316361 13017 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:02:55.316471 13017 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:55.317476 13017 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } }
I20260131 03:02:55.317817 13017 leader_election.cc:304] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2abf73bb02194843aaeb35f43d12cf52; no voters:
I20260131 03:02:55.318104 13017 leader_election.cc:290] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:02:55.318495 13022 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:55.318692 13022 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 1 LEADER]: Becoming Leader. State: Replica: 2abf73bb02194843aaeb35f43d12cf52, State: Running, Role: LEADER
I20260131 03:02:55.318946 13022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } }
I20260131 03:02:55.319255 13017 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:55.320297 13023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } }
I20260131 03:02:55.320415 13023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:55.320700 13024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2abf73bb02194843aaeb35f43d12cf52. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } }
I20260131 03:02:55.320780 13024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:55.320853 13031 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:02:55.324692 13031 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:02:55.330315 13031 catalog_manager.cc:1357] Generated new cluster ID: 36b79362917a44fb8ac3b28dff189dc6
I20260131 03:02:55.330402 13031 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:02:55.360361 13031 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:02:55.361287 13031 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:02:55.366947 13031 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52: Generated new TSK 0
I20260131 03:02:55.367591 13031 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
I20260131 03:02:55.426072 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:36467
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:55.542332 13045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:55.542511 13045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:55.542560 13045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:55.545408 13045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:55.545465 13045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:55.545542 13045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:02:55.549259 13045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:55.550175 13045 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:55.550900 13045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:55.557001 13051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:55.557047 13053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:55.557047 13050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:55.557412 13045 server_base.cc:1060] running on GCE node
I20260131 03:02:55.557765 13045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:55.558274 13045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:55.559484 13045 hybrid_clock.cc:648] HybridClock initialized: now 1769828575559444 us; error 53 us; skew 500 ppm
Jan 31 03:02:55 dist-test-slave-j368 krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:55.561941 13045 init.cc:377] Logged in from keytab as oryx/127.6.22.65@KRBTEST.COM (short username oryx)
I20260131 03:02:55.562835 13045 webserver.cc:492] Webserver started at http://127.6.22.65:45325/ using document root <none> and password file <none>
I20260131 03:02:55.563311 13045 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:55.563355 13045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:55.563540 13045 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:55.565634 13045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "dec07f75ac0344b89969f8353b215e19"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "e7c28272f713c3e71678d74383577ee9"
server_key_iv: "d26f4d3b26f4e3b425648da2f695a4a1"
server_key_version: "encryptionkey@0"
I20260131 03:02:55.566546 13045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "dec07f75ac0344b89969f8353b215e19"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "e7c28272f713c3e71678d74383577ee9"
server_key_iv: "d26f4d3b26f4e3b425648da2f695a4a1"
server_key_version: "encryptionkey@0"
I20260131 03:02:55.569726 13045 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260131 03:02:55.572009 13060 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:55.573089 13045 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:55.573238 13045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "dec07f75ac0344b89969f8353b215e19"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "e7c28272f713c3e71678d74383577ee9"
server_key_iv: "d26f4d3b26f4e3b425648da2f695a4a1"
server_key_version: "encryptionkey@0"
I20260131 03:02:55.573366 13045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:55.604568 13045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:55.607133 13045 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:55.607344 13045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:55.607909 13045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:55.608789 13045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:55.608878 13045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:55.608953 13045 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:55.609023 13045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:55.618522 13045 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:40815
I20260131 03:02:55.618551 13173 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:40815 every 8 connection(s)
I20260131 03:02:55.619436 13045 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:02:55.622759 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 13045
I20260131 03:02:55.622879 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:02:55.623143 6233 external_mini_cluster.cc:1468] Setting key cde8a858dd39e9cd3c52fd69a97d54c3
Jan 31 03:02:55 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.6.22.66@KRBTEST.COM; defaulting to no policy
I20260131 03:02:55.633363 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:55.621432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:49559 (local address 127.6.22.126:36467)
0131 03:02:55.621584 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:55.621589 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:55.622236 (+ 647us) server_negotiation.cc:408] Connection header received
0131 03:02:55.623103 (+ 867us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:55.623106 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:55.623148 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:55.623213 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:55.625324 (+ 2111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:55.626005 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:55.626808 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:55.627004 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:55.629331 (+ 2327us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:55.629376 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:55.629382 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:55.629418 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:55.631100 (+ 1682us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:55.631648 (+ 548us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:55.631654 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:55.631659 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:55.631724 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:55.632031 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:55.632036 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:55.632038 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:55.632286 (+ 248us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:55.632412 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:55.632988 (+ 576us) server_negotiation.cc:300] Negotiation successful
0131 03:02:55.633145 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260131 03:02:55.633848 13176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:55.621700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.65:49559)
0131 03:02:55.622090 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:55.622122 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:55.622909 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:55.623384 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:55.623393 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:55.623888 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:55.625045 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:55.625065 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:55.626261 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:55.626264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:55.626677 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:55.626685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:55.626858 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:55.627489 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:55.627510 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:55.629173 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:55.631276 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:55.631281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:55.631290 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:55.631513 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:55.631833 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:55.631836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:55.631837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:55.631925 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:55.632403 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:55.632408 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:55.632630 (+ 222us) client_negotiation.cc:769] Sending connection context
0131 03:02:55.632896 (+ 266us) client_negotiation.cc:240] Negotiation successful
0131 03:02:55.633125 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
I20260131 03:02:55.635000 13174 heartbeater.cc:344] Connected to a master server at 127.6.22.126:36467
I20260131 03:02:55.635259 13174 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:55.635833 13174 heartbeater.cc:507] Master 127.6.22.126:36467 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.66@KRBTEST.COM" created.
I20260131 03:02:55.637591 12981 ts_manager.cc:194] Registered new tserver with Master: dec07f75ac0344b89969f8353b215e19 (127.6.22.65:40815)
I20260131 03:02:55.638952 12981 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:49559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
I20260131 03:02:55.671835 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:36467
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:55.770740 13181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:55.770957 13181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:55.771042 13181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:55.774179 13181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:55.774267 13181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:55.774379 13181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:02:55.778330 13181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:55.779243 13181 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:55.779994 13181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:55.785856 13189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:55.785856 13187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:55.785856 13186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:55.786084 13181 server_base.cc:1060] running on GCE node
I20260131 03:02:55.786489 13181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:55.786965 13181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:55.788128 13181 hybrid_clock.cc:648] HybridClock initialized: now 1769828575788101 us; error 49 us; skew 500 ppm
Jan 31 03:02:55 dist-test-slave-j368 krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:55.790534 13181 init.cc:377] Logged in from keytab as oryx/127.6.22.66@KRBTEST.COM (short username oryx)
I20260131 03:02:55.791392 13181 webserver.cc:492] Webserver started at http://127.6.22.66:40681/ using document root <none> and password file <none>
I20260131 03:02:55.791814 13181 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:55.791853 13181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:55.791977 13181 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:55.793437 13181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "256c3d201c82434cb944e16db43d2674"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "f2a0ac2eaa124f40c5cbcf8085856fa9"
server_key_iv: "4576c9f9ee433613e9102ab50ea07251"
server_key_version: "encryptionkey@0"
I20260131 03:02:55.793856 13181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "256c3d201c82434cb944e16db43d2674"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "f2a0ac2eaa124f40c5cbcf8085856fa9"
server_key_iv: "4576c9f9ee433613e9102ab50ea07251"
server_key_version: "encryptionkey@0"
I20260131 03:02:55.796849 13181 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260131 03:02:55.798781 13196 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:55.799634 13181 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:55.799741 13181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "256c3d201c82434cb944e16db43d2674"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "f2a0ac2eaa124f40c5cbcf8085856fa9"
server_key_iv: "4576c9f9ee433613e9102ab50ea07251"
server_key_version: "encryptionkey@0"
I20260131 03:02:55.799821 13181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:55.812773 13181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:55.815007 13181 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:55.815219 13181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:55.815713 13181 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:55.816529 13181 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:55.816568 13181 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:55.816632 13181 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:55.816720 13181 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:55.825524 13181 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:44517
I20260131 03:02:55.825542 13309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:44517 every 8 connection(s)
I20260131 03:02:55.826604 13181 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:02:55 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:02:55.836381 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 13181
I20260131 03:02:55.836479 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:02:55.836678 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:55.828102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:38177 (local address 127.6.22.126:36467)
0131 03:02:55.828246 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:55.828250 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:55.828891 (+ 641us) server_negotiation.cc:408] Connection header received
0131 03:02:55.829533 (+ 642us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:55.829536 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:55.829577 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:55.829642 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:55.830970 (+ 1328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:55.831385 (+ 415us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:55.831889 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:55.831993 (+ 104us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:55.833762 (+ 1769us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:55.833777 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:55.833779 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:55.833800 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:55.835091 (+ 1291us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:55.835494 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:55.835496 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:55.835498 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:55.835537 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:55.835782 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:55.835784 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:55.835785 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:55.835910 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:55.835982 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:55.836388 (+ 406us) server_negotiation.cc:300] Negotiation successful
0131 03:02:55.836510 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260131 03:02:55.836737 6233 external_mini_cluster.cc:1468] Setting key d88a86048038656aefe1e5aaafaf4583
I20260131 03:02:55.837199 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:55.828347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.66:38177)
0131 03:02:55.828782 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:55.828812 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:55.829384 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:55.829770 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:55.829776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:55.830068 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:55.830858 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:55.830870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:55.831482 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:55.831485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:55.831803 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:55.831809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:55.831941 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:55.832367 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:55.832386 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:55.833640 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:55.835191 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:55.835195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:55.835202 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:55.835401 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:55.835619 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:55.835621 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:55.835623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:55.835705 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:55.835997 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:55.836001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:55.836176 (+ 175us) client_negotiation.cc:769] Sending connection context
0131 03:02:55.836311 (+ 135us) client_negotiation.cc:240] Negotiation successful
0131 03:02:55.836530 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260131 03:02:55.838197 13310 heartbeater.cc:344] Connected to a master server at 127.6.22.126:36467
I20260131 03:02:55.838428 13310 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:55.838940 13310 heartbeater.cc:507] Master 127.6.22.126:36467 requested a full tablet report, sending...
I20260131 03:02:55.839918 12981 ts_manager.cc:194] Registered new tserver with Master: 256c3d201c82434cb944e16db43d2674 (127.6.22.66:44517)
I20260131 03:02:55.840552 12981 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:38177
WARNING: no policy specified for oryx/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
I20260131 03:02:55.876948 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:36467
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:55.968297 13317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:55.968513 13317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:55.968590 13317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:55.971315 13317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:55.971407 13317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:55.971525 13317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:02:55.974987 13317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:55.975898 13317 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:55.976635 13317 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:55.982579 13322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:55.982569 13323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:55.982687 13317 server_base.cc:1060] running on GCE node
W20260131 03:02:55.982568 13325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:55.983104 13317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:55.983546 13317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:55.984714 13317 hybrid_clock.cc:648] HybridClock initialized: now 1769828575984693 us; error 36 us; skew 500 ppm
Jan 31 03:02:55 dist-test-slave-j368 krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:55.987001 13317 init.cc:377] Logged in from keytab as oryx/127.6.22.67@KRBTEST.COM (short username oryx)
I20260131 03:02:55.987799 13317 webserver.cc:492] Webserver started at http://127.6.22.67:43837/ using document root <none> and password file <none>
I20260131 03:02:55.988221 13317 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:55.988266 13317 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:55.988415 13317 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:55.989893 13317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "35f40d5fb02a49a6828f298950421a75"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "d6d279540fedabf9370c175c91bbb32f"
server_key_iv: "2d028e7f91e93b35ed910025766546ac"
server_key_version: "encryptionkey@0"
I20260131 03:02:55.990269 13317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "35f40d5fb02a49a6828f298950421a75"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "d6d279540fedabf9370c175c91bbb32f"
server_key_iv: "2d028e7f91e93b35ed910025766546ac"
server_key_version: "encryptionkey@0"
I20260131 03:02:55.993563 13317 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260131 03:02:55.995381 13332 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:55.996279 13317 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:55.996392 13317 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "35f40d5fb02a49a6828f298950421a75"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "d6d279540fedabf9370c175c91bbb32f"
server_key_iv: "2d028e7f91e93b35ed910025766546ac"
server_key_version: "encryptionkey@0"
I20260131 03:02:55.996484 13317 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:56.011927 13317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:56.014760 13317 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:56.015014 13317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:56.015617 13317 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:56.016512 13317 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:56.016556 13317 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:56.016587 13317 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:56.016599 13317 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:56.024140 13317 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:37933
I20260131 03:02:56.024295 13445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:37933 every 8 connection(s)
I20260131 03:02:56.025014 13317 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:02:56.031731 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 13317
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:02:56.031840 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:02:56.032164 6233 external_mini_cluster.cc:1468] Setting key fcf8537e25c781d31d263d76bb919905
I20260131 03:02:56.035746 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.026603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:58485 (local address 127.6.22.126:36467)
0131 03:02:56.026738 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.026741 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.027280 (+ 539us) server_negotiation.cc:408] Connection header received
0131 03:02:56.027908 (+ 628us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.027911 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.027952 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.028018 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.029337 (+ 1319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.029753 (+ 416us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.030282 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.030399 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.032458 (+ 2059us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.032479 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.032486 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.032521 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.034216 (+ 1695us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.034607 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.034609 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.034611 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.034650 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.034907 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.034909 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.034911 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.035045 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.035124 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.035530 (+ 406us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.035635 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260131 03:02:56.036315 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.026799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.67:58485)
0131 03:02:56.027149 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.027177 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.027756 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.028141 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.028147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.028431 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.029225 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.029237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.029844 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.029847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.030187 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.030192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.030348 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.030825 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.030845 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.032309 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.034314 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.034317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.034324 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.034520 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.034725 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.034728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.034729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.034808 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.035134 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.035138 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.035327 (+ 189us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.035478 (+ 151us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.035648 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
I20260131 03:02:56.037395 13446 heartbeater.cc:344] Connected to a master server at 127.6.22.126:36467
I20260131 03:02:56.037626 13446 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:56.038062 13446 heartbeater.cc:507] Master 127.6.22.126:36467 requested a full tablet report, sending...
I20260131 03:02:56.038944 12981 ts_manager.cc:194] Registered new tserver with Master: 35f40d5fb02a49a6828f298950421a75 (127.6.22.67:37933)
I20260131 03:02:56.039410 12981 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:58485
I20260131 03:02:56.045748 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
I20260131 03:02:56.084694 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.125:32917
--webserver_interface=127.6.22.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:36467,127.6.22.125:32917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:56.176005 13453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:56.176332 13453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:56.176445 13453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:56.180063 13453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:56.180188 13453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:56.180413 13453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:56.180554 13453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:56.180619 13453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:56.186472 13453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:36467,127.6.22.125:32917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.125:32917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.22.125
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
W20260131 03:02:56.186803 13453 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.22.126:36467,127.6.22.125:32917'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260131 03:02:56.188009 13453 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:56.189291 13453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:56.195844 13459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:56.195844 13458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:56.195847 13461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:56.196314 13453 server_base.cc:1060] running on GCE node
I20260131 03:02:56.196878 13453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:56.197793 13453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:56.199016 13453 hybrid_clock.cc:648] HybridClock initialized: now 1769828576198945 us; error 86 us; skew 500 ppm
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828576, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:56.201954 13453 init.cc:377] Logged in from keytab as oryx/127.6.22.125@KRBTEST.COM (short username oryx)
I20260131 03:02:56.203135 13453 webserver.cc:492] Webserver started at http://127.6.22.125:39849/ using document root <none> and password file <none>
I20260131 03:02:56.203795 13453 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:56.203853 13453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:56.204027 13453 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:56.205861 13453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data/instance:
uuid: "f4958e667bad4512a95edb0f583dee0a"
format_stamp: "Formatted at 2026-01-31 03:02:56 on dist-test-slave-j368"
server_key: "01c93236464faccda18fd2550a83c395"
server_key_iv: "f8a926a4d0020fd7c9c6a3525295bf14"
server_key_version: "encryptionkey@0"
I20260131 03:02:56.206362 13453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/wal/instance:
uuid: "f4958e667bad4512a95edb0f583dee0a"
format_stamp: "Formatted at 2026-01-31 03:02:56 on dist-test-slave-j368"
server_key: "01c93236464faccda18fd2550a83c395"
server_key_iv: "f8a926a4d0020fd7c9c6a3525295bf14"
server_key_version: "encryptionkey@0"
I20260131 03:02:56.210500 13453 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260131 03:02:56.213403 13468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:56.214859 13453 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:02:56.215024 13453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/wal
uuid: "f4958e667bad4512a95edb0f583dee0a"
format_stamp: "Formatted at 2026-01-31 03:02:56 on dist-test-slave-j368"
server_key: "01c93236464faccda18fd2550a83c395"
server_key_iv: "f8a926a4d0020fd7c9c6a3525295bf14"
server_key_version: "encryptionkey@0"
I20260131 03:02:56.215207 13453 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:56.227916 13453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:56.231683 13453 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:56.231974 13453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:56.240340 13520 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.125:32917 every 8 connection(s)
I20260131 03:02:56.240336 13453 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.125:32917
I20260131 03:02:56.241520 13453 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data/info.pb
I20260131 03:02:56.245606 13521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:56.249805 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 13453
I20260131 03:02:56.249953 13521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828576, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.125@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.22.125@KRBTEST.COM
I20260131 03:02:56.262301 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.252253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.6.22.126:36467)
0131 03:02:56.252387 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.252391 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.252597 (+ 206us) server_negotiation.cc:408] Connection header received
0131 03:02:56.252979 (+ 382us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.252982 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.253031 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.253143 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.254065 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.254578 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.255234 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.255408 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.258084 (+ 2676us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.258114 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.258121 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.258161 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.259833 (+ 1672us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.260337 (+ 504us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.260341 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.260344 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.260402 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.260871 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.260875 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.260877 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.261164 (+ 287us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.261245 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.261965 (+ 720us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.262107 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260131 03:02:56.263048 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.251495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58522 (local address 127.6.22.125:32917)
0131 03:02:56.251921 (+ 426us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.251929 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.251959 (+ 30us) server_negotiation.cc:408] Connection header received
0131 03:02:56.252736 (+ 777us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.252760 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.252978 (+ 218us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.253174 (+ 196us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.254128 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.254793 (+ 665us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.255454 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.255714 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.258566 (+ 2852us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.258593 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.258606 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.258640 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.260609 (+ 1969us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.261136 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.261142 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.261144 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.261202 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.261498 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.261501 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.261503 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.261744 (+ 241us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.261946 (+ 202us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.262199 (+ 253us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.262319 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":131,"threads_started":1}
I20260131 03:02:56.263048 13525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.252066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:35588)
0131 03:02:56.252450 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.252492 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.252797 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.253193 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.253198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.253606 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.253908 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.253921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.254700 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.254705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.255097 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.255104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.255328 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.256136 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.256159 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.257902 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.259970 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.259978 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.259985 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.260215 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.260555 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.260558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.260559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.260762 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.261270 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.261291 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.261650 (+ 359us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.261854 (+ 204us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.262181 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"spinlock_wait_cycles":1280,"thread_start_us":138,"threads_started":1}
I20260131 03:02:56.265029 13521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828576, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.125@KRBTEST.COM for oryx/127.6.22.125@KRBTEST.COM
I20260131 03:02:56.273442 13525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.265575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.125:32917 (local address 127.0.0.1:58530)
0131 03:02:56.265706 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.265720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.265799 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.266003 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.266005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.266164 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.266349 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.266355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.267298 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.267301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.267691 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.267698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.267799 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.268348 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.268362 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.269684 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.271742 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.271746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.271748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.271960 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.272281 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.272284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.272286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.272344 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.272837 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.272840 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.272945 (+ 105us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.273057 (+ 112us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.273255 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260131 03:02:56.273624 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.265664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.6.22.125:32917)
0131 03:02:56.265787 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.265790 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.265802 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:02:56.265868 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.265870 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.265907 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.265973 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.266571 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.267141 (+ 570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.267840 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.268011 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.269837 (+ 1826us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.269854 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.269856 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.269879 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.271513 (+ 1634us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.272104 (+ 591us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.272106 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.272108 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.272156 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.272453 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.272455 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.272456 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.272619 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.272821 (+ 202us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.273139 (+ 318us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.273359 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":45}
I20260131 03:02:56.276631 13521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: Bootstrap starting.
I20260131 03:02:56.279379 13521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:56.280285 13521 log.cc:826] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:56.282387 13521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: No bootstrap required, opened a new log
I20260131 03:02:56.284914 13521 raft_consensus.cc:359] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } }
I20260131 03:02:56.285115 13521 raft_consensus.cc:385] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:56.285205 13521 raft_consensus.cc:740] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4958e667bad4512a95edb0f583dee0a, State: Initialized, Role: FOLLOWER
I20260131 03:02:56.285653 13521 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } }
I20260131 03:02:56.286254 13527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } } }
I20260131 03:02:56.286424 13527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:02:56.286828 13521 sys_catalog.cc:565] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: configured and running, proceeding with master startup.
W20260131 03:02:56.291754 13538 catalog_manager.cc:1568] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:02:56.291865 13538 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:02:56.294262 13453 master_runner.cc:428] Detected that this master f4958e667bad4512a95edb0f583dee0a is joining an existing cluster
I20260131 03:02:56.294350 13453 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.22.125:32917
I20260131 03:02:56.306138 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.296716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.6.22.126:36467)
0131 03:02:56.296866 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.296873 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.297238 (+ 365us) server_negotiation.cc:408] Connection header received
0131 03:02:56.297415 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.297420 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.297480 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.297540 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.298739 (+ 1199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.299509 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.300804 (+ 1295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.301020 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.301789 (+ 769us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.301813 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.301817 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.301848 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.303948 (+ 2100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.304522 (+ 574us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.304527 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.304530 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.304594 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.304973 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.304977 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.304982 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.305217 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.305337 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.305827 (+ 490us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.305955 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260131 03:02:56.306150 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.296616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:35596)
0131 03:02:56.297100 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.297114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.297268 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.297693 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.297695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.297895 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.298599 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.298609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.299675 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.299680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.300336 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.300346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.300473 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.301198 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.301217 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.301640 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.304087 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.304093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.304099 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.304365 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.304761 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.304769 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.304774 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.304840 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.305386 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.305390 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.305504 (+ 114us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.305637 (+ 133us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.305923 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":188,"threads_started":1}
I20260131 03:02:56.308554 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.296832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58538 (local address 127.6.22.125:32917)
0131 03:02:56.296944 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.296949 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.298973 (+ 2024us) server_negotiation.cc:408] Connection header received
0131 03:02:56.299121 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.299124 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.299161 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.299223 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.299989 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.300586 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.301771 (+ 1185us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.301964 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.303734 (+ 1770us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.303754 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.303758 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.303789 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.305799 (+ 2010us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.307279 (+ 1480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.307285 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.307290 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.307344 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.307628 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.307634 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.307636 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.307862 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.307967 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.308259 (+ 292us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.308380 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260131 03:02:56.309094 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.296774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.125:32917 (local address 127.0.0.1:58538)
0131 03:02:56.298900 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.298916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.299027 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.299336 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.299339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.299590 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.299868 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.299875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.301088 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.301092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.301627 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.301638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.302423 (+ 785us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.303156 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.303181 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.303583 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.306894 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.306900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.306905 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.307131 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.307465 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.307471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.307473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.307528 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.308003 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.308006 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.308081 (+ 75us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.308774 (+ 693us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.308925 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2036,"thread_start_us":102,"threads_started":1}
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.126@KRBTEST.COM for oryx/127.6.22.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.6.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
I20260131 03:02:56.346891 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.312107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.125:32917 (local address 127.0.0.1:58550)
0131 03:02:56.317002 (+ 4895us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.317047 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.317251 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.317671 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.317677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.318212 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.318526 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.318535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.319449 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.319456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.319975 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.319986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.320158 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.320868 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.320891 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.322905 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.344706 (+ 21801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.344721 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.344729 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.345033 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.345402 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.345407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.345410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.345495 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.346046 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.346057 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.346211 (+ 154us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.346376 (+ 165us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.346599 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4686,"thread_start_us":131,"threads_started":1}
I20260131 03:02:56.347007 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.312124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.6.22.125:32917)
0131 03:02:56.312290 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.312295 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.317202 (+ 4907us) server_negotiation.cc:408] Connection header received
0131 03:02:56.317418 (+ 216us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.317425 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.317507 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.317610 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.318680 (+ 1070us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.319311 (+ 631us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.320105 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.320286 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.323079 (+ 2793us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.323106 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.323110 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.323145 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.344492 (+ 21347us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.345161 (+ 669us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.345166 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.345168 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.345265 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.345650 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.345655 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.345658 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.345897 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.346016 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.346498 (+ 482us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.346632 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260131 03:02:56.349156 12981 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.6.22.125:32917: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f4958e667bad4512a95edb0f583dee0a"
member_type: NON_VOTER
last_known_addr {
host: "127.6.22.125"
port: 32917
}
attrs {
promote: true
}
}
dest_uuid: "2abf73bb02194843aaeb35f43d12cf52"
cas_config_opid_index: -1
I20260131 03:02:56.350149 12981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true } }
I20260131 03:02:56.352620 13550 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f4958e667bad4512a95edb0f583dee0a (127.6.22.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true } } }
WARNING: no policy specified for HTTP/127.6.22.124@KRBTEST.COM; defaulting to no policy
I20260131 03:02:56.353662 13495 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:56.355314 13550 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.6.22.125:32917
I20260131 03:02:56.355645 13495 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 FOLLOWER]: Refusing update from remote peer 2abf73bb02194843aaeb35f43d12cf52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260131 03:02:56.355839 13550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true } } }
I20260131 03:02:56.355994 13550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.124@KRBTEST.COM" created.
I20260131 03:02:56.356698 13453 master.cc:561] Master@127.6.22.125:32917 shutting down...
I20260131 03:02:56.357137 13551 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260131 03:02:56.359426 12966 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260131 03:02:56.360491 12966 consensus_peers.cc:597] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 -> Peer f4958e667bad4512a95edb0f583dee0a (127.6.22.125:32917): Couldn't send request to peer f4958e667bad4512a95edb0f583dee0a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260131 03:02:56.362695 13453 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:56.362982 13453 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:56.363090 13453 tablet_replica.cc:333] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab.
I20260131 03:02:56.376421 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.124:41045
--webserver_interface=127.6.22.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:36467,127.6.22.125:32917,127.6.22.124:41045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:02:56.378662 13453 master.cc:583] Master@127.6.22.125:32917 shutdown complete.
I20260131 03:02:56.378747 13453 master_runner.cc:305] Clearing existing system tablet
I20260131 03:02:56.379009 13453 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:56.379072 13453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:56.380861 13453 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:02:56.382413 13557 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:56.382849 13453 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:56.383014 13453 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:02:56.383057 13453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/wal
uuid: "f4958e667bad4512a95edb0f583dee0a"
format_stamp: "Formatted at 2026-01-31 03:02:56 on dist-test-slave-j368"
server_key: "01c93236464faccda18fd2550a83c395"
server_key_iv: "f8a926a4d0020fd7c9c6a3525295bf14"
server_key_version: "encryptionkey@0"
I20260131 03:02:56.385279 13453 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:56.387671 13453 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260131 03:02:56.387835 13453 log.cc:1199] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260131 03:02:56.388166 13453 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: Deleting consensus metadata
I20260131 03:02:56.388361 13453 master_runner.cc:315] Copying system tablet from 127.6.22.126:36467
I20260131 03:02:56.389710 13453 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.22.126:36467
I20260131 03:02:56.399096 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.390455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.6.22.126:36467)
0131 03:02:56.390607 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.390613 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.390806 (+ 193us) server_negotiation.cc:408] Connection header received
0131 03:02:56.390925 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.390929 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.390990 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.391094 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.391928 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.392820 (+ 892us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.393727 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.393912 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.395465 (+ 1553us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.395495 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.395498 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.395526 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.397349 (+ 1823us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.397858 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.397861 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.397863 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.397914 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.398216 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.398221 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.398224 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.398412 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.398516 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.398817 (+ 301us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.398933 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260131 03:02:56.399123 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.390325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:35606)
0131 03:02:56.390695 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.390711 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.390830 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.391238 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.391242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.391474 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.391762 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.391771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.392950 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.392954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.393583 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.393594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.393727 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.394798 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.394824 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.395266 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.397487 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.397492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.397495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.397734 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.398028 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.398032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.398035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.398097 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.398521 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.398524 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.398628 (+ 104us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.398752 (+ 124us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.398896 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":94,"threads_started":1}
I20260131 03:02:56.400190 13001 tablet_copy_service.cc:140] P 2abf73bb02194843aaeb35f43d12cf52: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f4958e667bad4512a95edb0f583dee0a ({username='oryx', principal='oryx/127.6.22.125@KRBTEST.COM'} at 127.0.0.1:35606)
I20260131 03:02:56.400342 13001 tablet_copy_service.cc:161] P 2abf73bb02194843aaeb35f43d12cf52: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f4958e667bad4512a95edb0f583dee0a at {username='oryx', principal='oryx/127.6.22.125@KRBTEST.COM'} at 127.0.0.1:35606: session id = f4958e667bad4512a95edb0f583dee0a-00000000000000000000000000000000
I20260131 03:02:56.401962 13001 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52: Tablet Copy: opened 0 blocks and 1 log segments
I20260131 03:02:56.403622 13453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260131 03:02:56.406008 13453 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: tablet copy: Starting download of 0 data blocks...
I20260131 03:02:56.406189 13453 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: tablet copy: Starting download of 1 WAL segments...
I20260131 03:02:56.408161 13453 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260131 03:02:56.410161 13001 tablet_copy_service.cc:342] P 2abf73bb02194843aaeb35f43d12cf52: Request end of tablet copy session f4958e667bad4512a95edb0f583dee0a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.22.125@KRBTEST.COM'} at 127.0.0.1:35606
I20260131 03:02:56.410305 13001 tablet_copy_service.cc:434] P 2abf73bb02194843aaeb35f43d12cf52: ending tablet copy session f4958e667bad4512a95edb0f583dee0a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f4958e667bad4512a95edb0f583dee0a
W20260131 03:02:56.415280 13453 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260131 03:02:56.417755 13453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:56.419689 13564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:56.419911 13565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:56.420109 13453 server_base.cc:1060] running on GCE node
W20260131 03:02:56.420794 13567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:56.421079 13453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:56.421380 13453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:56.422537 13453 hybrid_clock.cc:648] HybridClock initialized: now 1769828576422519 us; error 48 us; skew 500 ppm
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828576, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:56.424338 13453 init.cc:377] Logged in from keytab as oryx/127.6.22.125@KRBTEST.COM (short username oryx)
I20260131 03:02:56.424911 13453 webserver.cc:492] Webserver started at http://127.6.22.125:44167/ using document root <none> and password file <none>
I20260131 03:02:56.425125 13453 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:56.425174 13453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:56.426661 13453 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:56.427796 13574 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:56.428135 13453 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:56.428225 13453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/wal
uuid: "f4958e667bad4512a95edb0f583dee0a"
format_stamp: "Formatted at 2026-01-31 03:02:56 on dist-test-slave-j368"
server_key: "01c93236464faccda18fd2550a83c395"
server_key_iv: "f8a926a4d0020fd7c9c6a3525295bf14"
server_key_version: "encryptionkey@0"
I20260131 03:02:56.428315 13453 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:56.440775 13453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:56.442957 13453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:56.449090 13453 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.125:32917
I20260131 03:02:56.449523 13626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.125:32917 every 8 connection(s)
I20260131 03:02:56.449795 13453 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-1/data/info.pb
I20260131 03:02:56.451570 13627 sys_catalog.cc:263] Verifying existing consensus state
I20260131 03:02:56.452527 13627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: Bootstrap starting.
I20260131 03:02:56.462395 13627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:02:56.462700 13627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: Bootstrap complete.
I20260131 03:02:56.463368 13627 raft_consensus.cc:359] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true } }
I20260131 03:02:56.463497 13627 raft_consensus.cc:740] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f4958e667bad4512a95edb0f583dee0a, State: Initialized, Role: LEARNER
I20260131 03:02:56.463599 13627 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true } }
I20260131 03:02:56.464051 13627 sys_catalog.cc:565] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:56.464869 13630 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true } } }
I20260131 03:02:56.465068 13630 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: This master's current role is: LEARNER
I20260131 03:02:56.468357 13641 catalog_manager.cc:1269] Loaded cluster ID: 36b79362917a44fb8ac3b28dff189dc6
I20260131 03:02:56.468429 13641 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: loading cluster ID for follower catalog manager: success
I20260131 03:02:56.470698 13641 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: acquiring CA information for follower catalog manager: success
I20260131 03:02:56.472118 13641 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260131 03:02:56.510298 13555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:56.510591 13555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:56.510701 13555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:56.514115 13555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:02:56.514222 13555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:56.514268 13555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:56.514308 13555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:02:56.514350 13555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:02:56.518844 13555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:36467,127.6.22.125:32917,127.6.22.124:41045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.124:41045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.22.124
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:56.520037 13555 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:56.521180 13555 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:56.527355 13647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:56.527421 13646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:56.527518 13649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:56.527868 13555 server_base.cc:1060] running on GCE node
I20260131 03:02:56.528417 13555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:56.529415 13555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:56.530663 13555 hybrid_clock.cc:648] HybridClock initialized: now 1769828576530632 us; error 54 us; skew 500 ppm
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828576, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:56.533789 13555 init.cc:377] Logged in from keytab as oryx/127.6.22.124@KRBTEST.COM (short username oryx)
I20260131 03:02:56.535075 13555 webserver.cc:492] Webserver started at http://127.6.22.124:34263/ using document root <none> and password file <none>
I20260131 03:02:56.535705 13555 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:56.535795 13555 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:56.536019 13555 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:02:56.537940 13555 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data/instance:
uuid: "54efaa3b58ae4b939212378c1ea9591e"
format_stamp: "Formatted at 2026-01-31 03:02:56 on dist-test-slave-j368"
server_key: "3baa39c9859ee5d49dc6182ae29f0696"
server_key_iv: "4f1004f96142f5932eb14625c5fe85ec"
server_key_version: "encryptionkey@0"
I20260131 03:02:56.538483 13555 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/wal/instance:
uuid: "54efaa3b58ae4b939212378c1ea9591e"
format_stamp: "Formatted at 2026-01-31 03:02:56 on dist-test-slave-j368"
server_key: "3baa39c9859ee5d49dc6182ae29f0696"
server_key_iv: "4f1004f96142f5932eb14625c5fe85ec"
server_key_version: "encryptionkey@0"
I20260131 03:02:56.542311 13555 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260131 03:02:56.545109 13656 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:56.546286 13555 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260131 03:02:56.546464 13555 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/wal
uuid: "54efaa3b58ae4b939212378c1ea9591e"
format_stamp: "Formatted at 2026-01-31 03:02:56 on dist-test-slave-j368"
server_key: "3baa39c9859ee5d49dc6182ae29f0696"
server_key_iv: "4f1004f96142f5932eb14625c5fe85ec"
server_key_version: "encryptionkey@0"
I20260131 03:02:56.546605 13555 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:56.562819 13555 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:56.566162 13555 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:56.566406 13555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:56.574560 13555 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.124:41045
I20260131 03:02:56.574556 13708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.124:41045 every 8 connection(s)
I20260131 03:02:56.575786 13555 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data/info.pb
I20260131 03:02:56.576032 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 13555
I20260131 03:02:56.579767 13709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:56.583575 13709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.22.124@KRBTEST.COM
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828576, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.124@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:02:56.590663 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.577630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.6.22.124:41045)
0131 03:02:56.578181 (+ 551us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.578191 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.578233 (+ 42us) server_negotiation.cc:408] Connection header received
0131 03:02:56.578988 (+ 755us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.579008 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.579339 (+ 331us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.579703 (+ 364us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.580786 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.581554 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.582339 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.582658 (+ 319us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.585200 (+ 2542us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.585253 (+ 53us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.585265 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.585308 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.587551 (+ 2243us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.588092 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.588097 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.588104 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.588207 (+ 103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.588548 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.588552 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.588554 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.589075 (+ 521us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.589274 (+ 199us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.589609 (+ 335us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.589872 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":137,"threads_started":1}
I20260131 03:02:56.594830 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.585511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.6.22.126:36467)
0131 03:02:56.585681 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.585684 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.585927 (+ 243us) server_negotiation.cc:408] Connection header received
0131 03:02:56.586185 (+ 258us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.586190 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.586262 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.586358 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.587281 (+ 923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.587946 (+ 665us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.588813 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.588956 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.591224 (+ 2268us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.591245 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.591247 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.591278 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.592994 (+ 1716us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.593524 (+ 530us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.593528 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.593531 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.593585 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.593869 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.593873 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.593875 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.594053 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.594172 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.594531 (+ 359us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.594656 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260131 03:02:56.594874 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.585337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:35612)
0131 03:02:56.585799 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.585849 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.586023 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.586408 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.586414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.586881 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.587118 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.587125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.588079 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.588083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.588695 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.588703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.588818 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.589421 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.589440 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.591070 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.593129 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.593138 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.593143 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.593397 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.593687 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.593691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.593693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.593768 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.594172 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.594181 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.594309 (+ 128us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.594465 (+ 156us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.594657 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":175,"threads_started":1}
I20260131 03:02:56.596087 13709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828576, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.124@KRBTEST.COM for oryx/127.6.22.125@KRBTEST.COM
I20260131 03:02:56.605253 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.596629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.125:32917 (local address 127.0.0.1:58566)
0131 03:02:56.596793 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.596807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.596887 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.597311 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.597314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.597507 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.597694 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.597700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.598652 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.598658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.599045 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.599053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.599190 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.599899 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.599912 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.601546 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.603802 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.603807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.603809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.604049 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.604309 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.604311 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.604313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.604361 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.604801 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.604803 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.604893 (+ 90us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.604984 (+ 91us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.605113 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260131 03:02:56.605289 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.596786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.6.22.125:32917)
0131 03:02:56.597092 (+ 306us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.597096 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.597109 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:02:56.597146 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.597148 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.597189 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.597298 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.597824 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.598524 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.599172 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.599385 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.601770 (+ 2385us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.601793 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.601795 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.601827 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.603631 (+ 1804us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.604157 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.604160 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.604162 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.604209 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.604445 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.604447 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.604448 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.604621 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.604780 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.605044 (+ 264us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.605157 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
I20260131 03:02:56.606200 13709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828576, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.124@KRBTEST.COM for oryx/127.6.22.124@KRBTEST.COM
I20260131 03:02:56.613701 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.606739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.124:41045 (local address 127.0.0.1:39732)
0131 03:02:56.606876 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.606892 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.606979 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.607192 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.607195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.607344 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.607523 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.607529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.608185 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.608187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.608539 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.608545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.608637 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.609216 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.609229 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.610491 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.612379 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.612382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.612384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.612541 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.612845 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.612848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.612849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.612893 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.613223 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.613225 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.613310 (+ 85us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.613395 (+ 85us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.613517 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260131 03:02:56.613706 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.606775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.6.22.124:41045)
0131 03:02:56.606906 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.606909 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.606957 (+ 48us) server_negotiation.cc:408] Connection header received
0131 03:02:56.607051 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.607054 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.607093 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.607160 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.607623 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.608085 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.608635 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.608849 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.610623 (+ 1774us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.610641 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.610644 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.610666 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.612257 (+ 1591us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.612629 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.612632 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.612634 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.612737 (+ 103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.612974 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.612976 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.612977 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.613139 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.613216 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.613450 (+ 234us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.613578 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":54}
I20260131 03:02:56.616958 13709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: Bootstrap starting.
I20260131 03:02:56.619895 13709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:56.620923 13709 log.cc:826] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:56.623176 13709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: No bootstrap required, opened a new log
I20260131 03:02:56.625901 13709 raft_consensus.cc:359] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } }
I20260131 03:02:56.626089 13709 raft_consensus.cc:385] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:56.626125 13709 raft_consensus.cc:740] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54efaa3b58ae4b939212378c1ea9591e, State: Initialized, Role: FOLLOWER
I20260131 03:02:56.626602 13709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } }
I20260131 03:02:56.627087 13715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } } }
I20260131 03:02:56.627211 13715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:02:56.627789 13709 sys_catalog.cc:565] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: configured and running, proceeding with master startup.
W20260131 03:02:56.633236 13726 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:02:56.633369 13726 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:02:56.637243 13555 master_runner.cc:428] Detected that this master 54efaa3b58ae4b939212378c1ea9591e is joining an existing cluster
I20260131 03:02:56.637357 13555 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.22.124:41045
I20260131 03:02:56.641012 13174 heartbeater.cc:499] Master 127.6.22.126:36467 was elected leader, sending a full tablet report...
I20260131 03:02:56.648782 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.644966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:35636)
0131 03:02:56.645179 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.645196 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.645283 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.645652 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.645655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.645676 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:56.645855 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.645859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.646955 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.646958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.647956 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.647986 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.648248 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.648333 (+ 85us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.648385 (+ 52us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.648476 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260131 03:02:56.649438 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.645062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.6.22.126:36467)
0131 03:02:56.645420 (+ 358us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.645424 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.645438 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:02:56.645490 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.645494 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.645543 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.645638 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:56.645974 (+ 336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.646839 (+ 865us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.648268 (+ 1429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.648986 (+ 718us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.649086 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.649155 (+ 69us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.649242 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":143,"threads_started":1}
I20260131 03:02:56.654242 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.650913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.125:32917 (local address 127.0.0.1:58578)
0131 03:02:56.651082 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.651094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.651194 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.651673 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.651677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.651714 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:56.651957 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.651964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.652820 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.652824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.653792 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.653803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.653937 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.653962 (+ 25us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.654015 (+ 53us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.654086 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260131 03:02:56.654868 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.651038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.6.22.125:32917)
0131 03:02:56.651418 (+ 380us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.651422 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.651433 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:02:56.651476 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.651478 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.651529 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.651614 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:56.652084 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.652615 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.653933 (+ 1318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.654265 (+ 332us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.654634 (+ 369us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.654682 (+ 48us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.654748 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":124,"threads_started":1}
I20260131 03:02:56.672235 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.639815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:35620)
0131 03:02:56.640723 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.640737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.641342 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.642321 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.642324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.665036 (+ 22712us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.665422 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.665434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.666560 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.666564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.667173 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.667182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.667305 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.667940 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.667957 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.668446 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.670615 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.670619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.670621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.670935 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.671200 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.671203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.671206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.671256 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.671688 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.671692 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.671779 (+ 87us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.671878 (+ 99us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.672020 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"spinlock_wait_cycles":7936,"thread_start_us":115,"threads_started":1}
I20260131 03:02:56.672848 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.639662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.6.22.126:36467)
0131 03:02:56.639794 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.639797 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.640825 (+ 1028us) server_negotiation.cc:408] Connection header received
0131 03:02:56.641483 (+ 658us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.641487 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.641545 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.641598 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.665615 (+ 24017us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.666422 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.667579 (+ 1157us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.667778 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.668608 (+ 830us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.668628 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.668632 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.668659 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.670475 (+ 1816us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.671050 (+ 575us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.671053 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.671056 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.671107 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.671363 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.671366 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.671369 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.671559 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.671647 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.672542 (+ 895us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.672700 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:02:56.673955 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.640389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.124:41045 (local address 127.0.0.1:39734)
0131 03:02:56.641357 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.641371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.641488 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.641785 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.641786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.664998 (+ 23212us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.665726 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.665736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.667313 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.667316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.667811 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.667824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.667934 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.668526 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.668542 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.668969 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.671289 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.671293 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.671296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.671556 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.672831 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.672834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.672836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.672878 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.673239 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.673244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.673333 (+ 89us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.673420 (+ 87us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.673715 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":79,"threads_started":1}
I20260131 03:02:56.674317 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.640468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.6.22.124:41045)
0131 03:02:56.640715 (+ 247us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.640719 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.641432 (+ 713us) server_negotiation.cc:408] Connection header received
0131 03:02:56.641586 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.641589 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.641627 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.641679 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.665862 (+ 24183us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.666519 (+ 657us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.668142 (+ 1623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.668370 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.669096 (+ 726us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.669116 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.669121 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.669146 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.671170 (+ 2024us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.672633 (+ 1463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.672640 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.672643 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.672721 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.672946 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.672949 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.672951 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.673119 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.673203 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.673572 (+ 369us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.674160 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":468,"server-negotiator.queue_time_us":133}
I20260131 03:02:56.675511 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.639686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.125:32917 (local address 127.0.0.1:58570)
0131 03:02:56.640024 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.640040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.640147 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.641064 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.641067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.666040 (+ 24973us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.666293 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.666301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.667136 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.667141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.668022 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.668033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.668396 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.670269 (+ 1873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.670292 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.670823 (+ 531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.673469 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.673472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.673474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.673725 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.674003 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.674006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.674008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.674058 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.674929 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.674933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.675023 (+ 90us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.675165 (+ 142us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.675297 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"spinlock_wait_cycles":46464,"thread_start_us":131,"threads_started":1}
I20260131 03:02:56.676069 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.640141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58570 (local address 127.6.22.125:32917)
0131 03:02:56.640279 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.640290 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.640303 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:02:56.640367 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.640370 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.640426 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.640507 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.666450 (+ 25943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.667011 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.668814 (+ 1803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.669308 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.670975 (+ 1667us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.671001 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.671005 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.671036 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.673346 (+ 2310us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.673834 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.673837 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.673840 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.673903 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.674642 (+ 739us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.674644 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.674646 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.674790 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.674937 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.675751 (+ 814us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.675894 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828575, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.126@KRBTEST.COM for oryx/127.6.22.124@KRBTEST.COM
I20260131 03:02:56.685797 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.677669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.6.22.124:41045)
0131 03:02:56.677783 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.677787 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.677800 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:02:56.677879 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.677884 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.677952 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.678037 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.678700 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.679143 (+ 443us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.679965 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.680091 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.682424 (+ 2333us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.682445 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.682447 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.682475 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.684085 (+ 1610us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.684604 (+ 519us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.684607 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.684608 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.684714 (+ 106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.684991 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.684993 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.684994 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.685158 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.685236 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.685563 (+ 327us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.685665 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260131 03:02:56.685842 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.677383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.124:41045 (local address 127.0.0.1:39740)
0131 03:02:56.677632 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.677647 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.677724 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.678090 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.678094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.678306 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.678558 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.678566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.679275 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.679278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.679840 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.679850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.679972 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.680517 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.680536 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.682243 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.684235 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.684239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.684242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.684493 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.684820 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.684824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.684826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.684894 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.685261 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.685270 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.685399 (+ 129us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.685513 (+ 114us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.685670 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260131 03:02:56.687024 12980 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.6.22.124:41045: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e"
member_type: NON_VOTER
last_known_addr {
host: "127.6.22.124"
port: 41045
}
attrs {
promote: true
}
}
dest_uuid: "2abf73bb02194843aaeb35f43d12cf52"
cas_config_opid_index: 5
I20260131 03:02:56.687711 12980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } }
I20260131 03:02:56.688853 13551 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 54efaa3b58ae4b939212378c1ea9591e (127.6.22.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:56.689445 13683 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:56.690500 13551 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.6.22.124:41045
I20260131 03:02:56.690850 13683 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 FOLLOWER]: Refusing update from remote peer 2abf73bb02194843aaeb35f43d12cf52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260131 03:02:56.691294 13555 master.cc:561] Master@127.6.22.124:41045 shutting down...
I20260131 03:02:56.691967 13551 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:56.692220 13550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:56.692339 13550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:56.692637 13601 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 LEARNER]: Refusing update from remote peer 2abf73bb02194843aaeb35f43d12cf52: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260131 03:02:56.693130 13737 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260131 03:02:56.694165 12967 consensus_peers.cc:597] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 -> Peer 54efaa3b58ae4b939212378c1ea9591e (127.6.22.124:41045): Couldn't send request to peer 54efaa3b58ae4b939212378c1ea9591e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260131 03:02:56.696092 13601 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 54efaa3b58ae4b939212378c1ea9591e (127.6.22.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:56.698886 13630 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2abf73bb02194843aaeb35f43d12cf52. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:56.699028 13630 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: This master's current role is: LEARNER
I20260131 03:02:56.699126 13630 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: NON_VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: true } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:56.699182 13630 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: This master's current role is: LEARNER
I20260131 03:02:56.699847 13555 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:56.700141 13555 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:56.700251 13555 tablet_replica.cc:333] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: stopping tablet replica
I20260131 03:02:56.700707 13738 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52: attempting to promote NON_VOTER f4958e667bad4512a95edb0f583dee0a to VOTER
I20260131 03:02:56.701301 13738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } }
I20260131 03:02:56.702893 13601 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 LEARNER]: Refusing update from remote peer 2abf73bb02194843aaeb35f43d12cf52: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260131 03:02:56.703337 13738 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260131 03:02:56.703883 12967 consensus_peers.cc:597] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 -> Peer 54efaa3b58ae4b939212378c1ea9591e (127.6.22.124:41045): Couldn't send request to peer 54efaa3b58ae4b939212378c1ea9591e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260131 03:02:56.705329 13738 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f4958e667bad4512a95edb0f583dee0a (127.6.22.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:56.706529 13601 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f4958e667bad4512a95edb0f583dee0a (127.6.22.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:56.707187 13550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:56.707324 13550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:56.707520 13550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:56.707605 13550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:56.707820 13555 master.cc:583] Master@127.6.22.124:41045 shutdown complete.
I20260131 03:02:56.707856 13555 master_runner.cc:305] Clearing existing system tablet
I20260131 03:02:56.708045 13555 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:56.708076 13555 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:56.709442 13555 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:02:56.710325 13741 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:56.710489 13741 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:02:56.710925 13743 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:56.711258 13555 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:56.711422 13555 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:56.711460 13555 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/wal
uuid: "54efaa3b58ae4b939212378c1ea9591e"
format_stamp: "Formatted at 2026-01-31 03:02:56 on dist-test-slave-j368"
server_key: "3baa39c9859ee5d49dc6182ae29f0696"
server_key_iv: "4f1004f96142f5932eb14625c5fe85ec"
server_key_version: "encryptionkey@0"
I20260131 03:02:56.714169 13555 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:56.716269 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.715952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.124:41045 (local address 127.0.0.1:39744)
0131 03:02:56.716100 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.716181 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.124:41045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260131 03:02:56.716538 12981 master.cc:679] Network error: unable to get registration information for peer 54efaa3b58ae4b939212378c1ea9591e: Client connection negotiation failed: client connection to 127.6.22.124:41045: connect: Connection refused (error 111)
I20260131 03:02:56.716869 13555 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260131 03:02:56.716974 13555 log.cc:1199] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260131 03:02:56.717305 13555 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: Deleting consensus metadata
I20260131 03:02:56.717496 13555 master_runner.cc:315] Copying system tablet from 127.6.22.126:36467
I20260131 03:02:56.718988 13555 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.22.126:36467
I20260131 03:02:56.728088 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.719786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.6.22.126:36467)
0131 03:02:56.719972 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.719977 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.720088 (+ 111us) server_negotiation.cc:408] Connection header received
0131 03:02:56.720229 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.720233 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.720307 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.720454 (+ 147us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:56.721207 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.722088 (+ 881us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.722851 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.723060 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.724383 (+ 1323us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:56.724411 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:56.724415 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:56.724442 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:56.726250 (+ 1808us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.726735 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.726738 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.726741 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.726794 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:56.727097 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:56.727102 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:56.727104 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:56.727356 (+ 252us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:56.727500 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.727781 (+ 281us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.727918 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260131 03:02:56.728111 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.719665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:35646)
0131 03:02:56.719974 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.719995 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.720112 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.720502 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.720505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.720807 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:56.721050 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.721057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.722245 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.722249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.722711 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.722719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.722843 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.723894 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:56.723910 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:56.724232 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:56.726383 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.726386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.726388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.726616 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.726898 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:56.726900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:56.726902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:56.726949 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:56.727482 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:56.727496 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:56.727597 (+ 101us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.727721 (+ 124us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.727903 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":142,"threads_started":1}
I20260131 03:02:56.728756 13001 tablet_copy_service.cc:140] P 2abf73bb02194843aaeb35f43d12cf52: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 54efaa3b58ae4b939212378c1ea9591e ({username='oryx', principal='oryx/127.6.22.124@KRBTEST.COM'} at 127.0.0.1:35646)
I20260131 03:02:56.728899 13001 tablet_copy_service.cc:161] P 2abf73bb02194843aaeb35f43d12cf52: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 54efaa3b58ae4b939212378c1ea9591e at {username='oryx', principal='oryx/127.6.22.124@KRBTEST.COM'} at 127.0.0.1:35646: session id = 54efaa3b58ae4b939212378c1ea9591e-00000000000000000000000000000000
I20260131 03:02:56.729910 13001 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52: Tablet Copy: opened 0 blocks and 1 log segments
I20260131 03:02:56.731745 13555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260131 03:02:56.734642 13555 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: tablet copy: Starting download of 0 data blocks...
I20260131 03:02:56.734877 13555 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: tablet copy: Starting download of 1 WAL segments...
I20260131 03:02:56.737123 13555 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260131 03:02:56.739571 13001 tablet_copy_service.cc:342] P 2abf73bb02194843aaeb35f43d12cf52: Request end of tablet copy session 54efaa3b58ae4b939212378c1ea9591e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.22.124@KRBTEST.COM'} at 127.0.0.1:35646
I20260131 03:02:56.739711 13001 tablet_copy_service.cc:434] P 2abf73bb02194843aaeb35f43d12cf52: ending tablet copy session 54efaa3b58ae4b939212378c1ea9591e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 54efaa3b58ae4b939212378c1ea9591e
W20260131 03:02:56.745898 13555 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260131 03:02:56.749244 13555 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:56.751271 13751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:56.751457 13753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:56.751885 13750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:56.751914 13555 server_base.cc:1060] running on GCE node
I20260131 03:02:56.752233 13555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:56.752585 13555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:56.753732 13555 hybrid_clock.cc:648] HybridClock initialized: now 1769828576753717 us; error 36 us; skew 500 ppm
I20260131 03:02:56.754478 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.754039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.124:41045 (local address 127.0.0.1:39752)
0131 03:02:56.754286 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.754357 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.124:41045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260131 03:02:56.754736 12981 master.cc:679] Network error: unable to get registration information for peer 54efaa3b58ae4b939212378c1ea9591e: Client connection negotiation failed: client connection to 127.6.22.124:41045: connect: Connection refused (error 111)
Jan 31 03:02:56 dist-test-slave-j368 krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828576, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:56.755972 13555 init.cc:377] Logged in from keytab as oryx/127.6.22.124@KRBTEST.COM (short username oryx)
I20260131 03:02:56.756567 13555 webserver.cc:492] Webserver started at http://127.6.22.124:35611/ using document root <none> and password file <none>
I20260131 03:02:56.756822 13555 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:56.756868 13555 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:56.758390 13555 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:56.759644 13760 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:56.760058 13555 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:02:56.760145 13555 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/wal
uuid: "54efaa3b58ae4b939212378c1ea9591e"
format_stamp: "Formatted at 2026-01-31 03:02:56 on dist-test-slave-j368"
server_key: "3baa39c9859ee5d49dc6182ae29f0696"
server_key_iv: "4f1004f96142f5932eb14625c5fe85ec"
server_key_version: "encryptionkey@0"
I20260131 03:02:56.760277 13555 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:56.769253 13555 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:56.771237 13555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:56.776789 13555 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.124:41045
I20260131 03:02:56.776888 13812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.124:41045 every 8 connection(s)
I20260131 03:02:56.777472 13555 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/master-2/data/info.pb
I20260131 03:02:56.779098 13813 sys_catalog.cc:263] Verifying existing consensus state
I20260131 03:02:56.780006 13813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: Bootstrap starting.
I20260131 03:02:56.790727 13813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:02:56.791116 13813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: Bootstrap complete.
I20260131 03:02:56.791731 13813 raft_consensus.cc:359] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } }
I20260131 03:02:56.791888 13813 raft_consensus.cc:740] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54efaa3b58ae4b939212378c1ea9591e, State: Initialized, Role: LEARNER
I20260131 03:02:56.792048 13813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } }
I20260131 03:02:56.792455 13816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:56.792567 13816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: This master's current role is: LEARNER
I20260131 03:02:56.792639 13813 sys_catalog.cc:565] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:02:56.797286 13827 catalog_manager.cc:1269] Loaded cluster ID: 36b79362917a44fb8ac3b28dff189dc6
I20260131 03:02:56.797360 13827 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: loading cluster ID for follower catalog manager: success
I20260131 03:02:56.799188 13827 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: acquiring CA information for follower catalog manager: success
I20260131 03:02:56.800797 13827 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260131 03:02:56.827025 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.823219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.124:41045 (local address 127.0.0.1:39758)
0131 03:02:56.823395 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:56.823409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:56.823486 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:56.824008 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:56.824012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:56.824041 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:56.824303 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.824314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.825565 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.825569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:56.826524 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:56.826537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.826684 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.826712 (+ 28us) client_negotiation.cc:769] Sending connection context
0131 03:02:56.826777 (+ 65us) client_negotiation.cc:240] Negotiation successful
0131 03:02:56.826864 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260131 03:02:56.828516 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:56.823416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.6.22.124:41045)
0131 03:02:56.823721 (+ 305us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:56.823725 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:56.823736 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:02:56.823778 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:56.823781 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:56.823831 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:56.823926 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:56.824452 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.825424 (+ 972us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:56.827358 (+ 1934us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:56.827929 (+ 571us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:56.828176 (+ 247us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:56.828247 (+ 71us) server_negotiation.cc:300] Negotiation successful
0131 03:02:56.828335 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":123,"threads_started":1}
I20260131 03:02:56.842747 13310 heartbeater.cc:499] Master 127.6.22.126:36467 was elected leader, sending a full tablet report...
I20260131 03:02:57.041227 13446 heartbeater.cc:499] Master 127.6.22.126:36467 was elected leader, sending a full tablet report...
I20260131 03:02:57.248450 13787 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260131 03:02:57.249047 13816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2abf73bb02194843aaeb35f43d12cf52. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:57.249229 13816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: This master's current role is: LEARNER
I20260131 03:02:57.772334 13829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: NON_VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: true } } }
I20260131 03:02:57.772565 13829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:57.773486 13831 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52: attempting to promote NON_VOTER 54efaa3b58ae4b939212378c1ea9591e to VOTER
I20260131 03:02:57.773903 13831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } }
I20260131 03:02:57.775388 13601 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 FOLLOWER]: Refusing update from remote peer 2abf73bb02194843aaeb35f43d12cf52: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260131 03:02:57.775487 13787 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 LEARNER]: Refusing update from remote peer 2abf73bb02194843aaeb35f43d12cf52: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260131 03:02:57.775942 13831 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260131 03:02:57.776269 13830 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260131 03:02:57.778580 13831 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 54efaa3b58ae4b939212378c1ea9591e (127.6.22.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } } }
I20260131 03:02:57.779233 13601 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 54efaa3b58ae4b939212378c1ea9591e (127.6.22.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } } }
I20260131 03:02:57.780237 13830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } } }
I20260131 03:02:57.780248 13831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } } }
I20260131 03:02:57.780377 13830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:57.780385 13831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:57.781915 13830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } } }
I20260131 03:02:57.782032 13830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52 [sys.catalog]: This master's current role is: LEADER
I20260131 03:02:57.782503 13787 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 54efaa3b58ae4b939212378c1ea9591e (127.6.22.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } } }
I20260131 03:02:57.784377 13837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } } }
I20260131 03:02:57.784550 13837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:02:57.785044 13836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2abf73bb02194843aaeb35f43d12cf52" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } } }
I20260131 03:02:57.785183 13836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:02:58.759171 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.751183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.6.22.125:32917)
0131 03:02:58.751464 (+ 281us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:58.751468 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:58.751480 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:02:58.751517 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:58.751520 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:58.751560 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:58.751650 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:58.752449 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.753004 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.753605 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.753743 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.755018 (+ 1275us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:58.755036 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:58.755039 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:58.755066 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:58.756906 (+ 1840us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:58.757464 (+ 558us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:58.757470 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:58.757471 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:58.757575 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:58.757946 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:58.757950 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:58.757951 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:58.758197 (+ 246us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:58.758385 (+ 188us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:58.758666 (+ 281us) server_negotiation.cc:300] Negotiation successful
0131 03:02:58.758921 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":113,"threads_started":1}
I20260131 03:02:58.763844 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.760482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:35650)
0131 03:02:58.760903 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:58.760925 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:58.761078 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:58.761476 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:58.761480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:58.761501 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:58.761808 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:58.761815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.762712 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.762718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:58.763364 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:58.763372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.763526 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.763551 (+ 25us) client_negotiation.cc:769] Sending connection context
0131 03:02:58.763606 (+ 55us) client_negotiation.cc:240] Negotiation successful
0131 03:02:58.763685 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":121,"threads_started":1}
I20260131 03:02:58.764345 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.760532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.6.22.126:36467)
0131 03:02:58.760866 (+ 334us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:58.760870 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:58.761055 (+ 185us) server_negotiation.cc:408] Connection header received
0131 03:02:58.761232 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:58.761236 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:58.761309 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:58.761438 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:58.761930 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.762588 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.763503 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.764019 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.764095 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:58.764149 (+ 54us) server_negotiation.cc:300] Negotiation successful
0131 03:02:58.764221 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":75,"threads_started":1}
I20260131 03:02:58.769232 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.766304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.125:32917 (local address 127.0.0.1:58592)
0131 03:02:58.766613 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:58.766641 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:58.766768 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:58.767024 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:58.767027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:58.767050 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:58.767276 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:58.767282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.768144 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.768148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:58.768798 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:58.768806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.768934 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.768958 (+ 24us) client_negotiation.cc:769] Sending connection context
0131 03:02:58.769014 (+ 56us) client_negotiation.cc:240] Negotiation successful
0131 03:02:58.769076 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"spinlock_wait_cycles":1664}
I20260131 03:02:58.769605 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.766392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.6.22.125:32917)
0131 03:02:58.766614 (+ 222us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:58.766619 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:58.766710 (+ 91us) server_negotiation.cc:408] Connection header received
0131 03:02:58.766881 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:58.766884 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:58.766926 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:58.767017 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:58.767397 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.768027 (+ 630us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.768921 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.769347 (+ 426us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.769395 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:58.769446 (+ 51us) server_negotiation.cc:300] Negotiation successful
0131 03:02:58.769511 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"spinlock_wait_cycles":4096}
I20260131 03:02:58.773241 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.770828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.124:41045 (local address 127.0.0.1:39762)
0131 03:02:58.770974 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:58.770986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:58.771047 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:58.771438 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:58.771441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:58.771453 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:58.771599 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:58.771603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.772310 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.772312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:58.772942 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:58.772951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.773046 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.773062 (+ 16us) client_negotiation.cc:769] Sending connection context
0131 03:02:58.773099 (+ 37us) client_negotiation.cc:240] Negotiation successful
0131 03:02:58.773144 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260131 03:02:58.773723 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.770966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.6.22.124:41045)
0131 03:02:58.771227 (+ 261us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:58.771230 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:58.771243 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:02:58.771289 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:58.771291 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:58.771335 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:58.771411 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:58.771705 (+ 294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.772210 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.773053 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.773469 (+ 416us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.773508 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:58.773566 (+ 58us) server_negotiation.cc:300] Negotiation successful
0131 03:02:58.773623 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":120,"threads_started":1}
I20260131 03:02:58.785054 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.775806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.6.22.124:41045)
0131 03:02:58.775937 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:58.775942 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:58.776000 (+ 58us) server_negotiation.cc:408] Connection header received
0131 03:02:58.776142 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:58.776145 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:58.776200 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:58.776291 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:58.777385 (+ 1094us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.778017 (+ 632us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.778928 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.779079 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.780551 (+ 1472us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:58.780581 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:58.780583 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:58.780612 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:58.782859 (+ 2247us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:58.783473 (+ 614us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:58.783476 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:58.783478 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:58.783525 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:58.783835 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:58.783838 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:58.783839 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:58.784023 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:58.784151 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:58.784762 (+ 611us) server_negotiation.cc:300] Negotiation successful
0131 03:02:58.784911 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260131 03:02:58.789002 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.786111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:35664)
0131 03:02:58.786515 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:58.786530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:58.786646 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:58.786949 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:58.786951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:58.786965 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:58.787174 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:58.787179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.787931 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.787934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:58.788543 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:58.788552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.788739 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.788763 (+ 24us) client_negotiation.cc:769] Sending connection context
0131 03:02:58.788811 (+ 48us) client_negotiation.cc:240] Negotiation successful
0131 03:02:58.788870 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":170,"threads_started":1}
I20260131 03:02:58.789441 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.786340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.6.22.126:36467)
0131 03:02:58.786521 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:58.786525 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:58.786708 (+ 183us) server_negotiation.cc:408] Connection header received
0131 03:02:58.786782 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:58.786784 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:58.786844 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:58.786926 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:58.787284 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.787823 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.788775 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.789166 (+ 391us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.789217 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:58.789269 (+ 52us) server_negotiation.cc:300] Negotiation successful
0131 03:02:58.789337 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260131 03:02:58.793462 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.790849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.125:32917 (local address 127.0.0.1:58608)
0131 03:02:58.791116 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:58.791130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:58.791295 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:58.791575 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:58.791581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:58.791605 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:58.791764 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:58.791769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.792453 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.792456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:58.793099 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:58.793107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.793248 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.793266 (+ 18us) client_negotiation.cc:769] Sending connection context
0131 03:02:58.793303 (+ 37us) client_negotiation.cc:240] Negotiation successful
0131 03:02:58.793349 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260131 03:02:58.794341 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.791079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.6.22.125:32917)
0131 03:02:58.791223 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:58.791227 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:58.791239 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:02:58.791400 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:58.791403 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:58.791446 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:58.791510 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:58.791879 (+ 369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.792346 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.793422 (+ 1076us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.794043 (+ 621us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.794091 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:58.794146 (+ 55us) server_negotiation.cc:300] Negotiation successful
0131 03:02:58.794205 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260131 03:02:58.798200 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.795920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.124:41045 (local address 127.0.0.1:39774)
0131 03:02:58.796073 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:58.796092 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:58.796166 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:58.796374 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:58.796376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:58.796389 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:58.796563 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:58.796568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.797337 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.797339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:58.797897 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:58.797904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.797995 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.798013 (+ 18us) client_negotiation.cc:769] Sending connection context
0131 03:02:58.798050 (+ 37us) client_negotiation.cc:240] Negotiation successful
0131 03:02:58.798095 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260131 03:02:58.798594 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.796020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.6.22.124:41045)
0131 03:02:58.796137 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:58.796140 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:58.796153 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:02:58.796249 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:58.796251 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:58.796289 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:58.796353 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:58.796698 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.797240 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:58.798001 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:58.798352 (+ 351us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:58.798413 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:58.798457 (+ 44us) server_negotiation.cc:300] Negotiation successful
0131 03:02:58.798508 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260131 03:02:58.800395 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 13045
I20260131 03:02:58.807534 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:40815
--local_ip_for_outbound_sockets=127.6.22.65
--tserver_master_addrs=127.6.22.126:36467
--webserver_port=45325
--webserver_interface=127.6.22.65
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.22.126:36467,127.6.22.125:32917
--tserver_master_addrs=127.6.22.126:36467,127.6.22.125:32917,127.6.22.124:41045 with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:02:58.929469 13845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:58.929792 13845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:58.929864 13845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:58.933480 13845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:58.933558 13845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:58.933681 13845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:02:58.938215 13845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:40815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=45325
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:36467,127.6.22.125:32917,127.6.22.124:41045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:58.939324 13845 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:58.940133 13845 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:58.947299 13853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:58.947311 13850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:58.947366 13851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:58.947541 13845 server_base.cc:1060] running on GCE node
I20260131 03:02:58.947926 13845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:58.948515 13845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:58.949692 13845 hybrid_clock.cc:648] HybridClock initialized: now 1769828578949667 us; error 41 us; skew 500 ppm
Jan 31 03:02:58 dist-test-slave-j368 krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828578, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:58.952561 13845 init.cc:377] Logged in from keytab as oryx/127.6.22.65@KRBTEST.COM (short username oryx)
I20260131 03:02:58.953686 13845 webserver.cc:492] Webserver started at http://127.6.22.65:45325/ using document root <none> and password file <none>
I20260131 03:02:58.954209 13845 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:58.954267 13845 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:58.957754 13845 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260131 03:02:58.959540 13860 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:58.960609 13845 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260131 03:02:58.960803 13845 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "dec07f75ac0344b89969f8353b215e19"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "e7c28272f713c3e71678d74383577ee9"
server_key_iv: "d26f4d3b26f4e3b425648da2f695a4a1"
server_key_version: "encryptionkey@0"
I20260131 03:02:58.961247 13845 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:58.978458 13845 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:58.981427 13845 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:58.981599 13845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:58.982184 13845 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:58.983129 13845 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:58.983212 13845 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:58.983281 13845 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:58.983336 13845 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:58.993420 13845 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:40815
I20260131 03:02:58.993440 13973 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:40815 every 8 connection(s)
I20260131 03:02:58.994606 13845 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:02:58.994978 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 13845
I20260131 03:02:58.995172 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 13181
I20260131 03:02:59.005641 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:44517
--local_ip_for_outbound_sockets=127.6.22.66
--tserver_master_addrs=127.6.22.126:36467
--webserver_port=40681
--webserver_interface=127.6.22.66
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.22.126:36467,127.6.22.125:32917
--tserver_master_addrs=127.6.22.126:36467,127.6.22.125:32917,127.6.22.124:41045 with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:02:59 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828578, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.125@KRBTEST.COM
Jan 31 03:02:59 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828578, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
Jan 31 03:02:59 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828578, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.124@KRBTEST.COM
I20260131 03:02:59.011916 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.996750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:58317 (local address 127.6.22.125:32917)
0131 03:02:58.997899 (+ 1149us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:58.997903 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:58.997919 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:02:58.999080 (+ 1161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:58.999085 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:58.999165 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:58.999497 (+ 332us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.001875 (+ 2378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.002376 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.003471 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.003654 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.006921 (+ 3267us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.006952 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.006957 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.007000 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.009112 (+ 2112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.009883 (+ 771us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.009888 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.009890 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.009945 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.010326 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.010332 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.010335 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.010591 (+ 256us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.010702 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.011503 (+ 801us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.011670 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059}
I20260131 03:02:59.015079 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.996913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.125:32917 (local address 127.6.22.65:58317)
0131 03:02:58.997653 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:58.997695 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:58.998754 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:58.999717 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:58.999731 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.000341 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:59.001670 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.001691 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.002581 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.002588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.003290 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.003304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.003559 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.004457 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:59.004502 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:59.006696 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:59.009261 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.009271 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.009288 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.009752 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.010066 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.010073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.010078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.010208 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.010781 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:59.010791 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:59.011158 (+ 367us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.013727 (+ 2569us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.014077 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"spinlock_wait_cycles":65408,"thread_start_us":119,"threads_started":1}
I20260131 03:02:59.016404 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.997251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:54713 (local address 127.6.22.126:36467)
0131 03:02:58.997404 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:58.997409 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.006075 (+ 8666us) server_negotiation.cc:408] Connection header received
0131 03:02:59.006290 (+ 215us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.006294 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.006357 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.006414 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.007254 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.008090 (+ 836us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.008883 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.009059 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.012281 (+ 3222us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.012303 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.012307 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.012344 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.014520 (+ 2176us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.015056 (+ 536us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.015062 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.015067 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.015125 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.015403 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.015407 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.015409 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.015626 (+ 217us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.015701 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.016065 (+ 364us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.016221 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260131 03:02:59.016889 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.997140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.65:54713)
0131 03:02:59.005968 (+ 8828us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.005982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.006150 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.006589 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.006592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.006796 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:59.007107 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.007115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.008233 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.008237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.008769 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.008777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.009166 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.009978 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:59.010004 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:59.012094 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:59.014679 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.014683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.014685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.014936 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.015236 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.015241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.015246 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.015305 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.015778 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:59.015781 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:59.015871 (+ 90us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.016553 (+ 682us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.016730 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8747,"thread_start_us":115,"threads_started":1}
I20260131 03:02:59.017262 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.997892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:34943 (local address 127.6.22.124:41045)
0131 03:02:58.998037 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:58.998041 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.005251 (+ 7210us) server_negotiation.cc:408] Connection header received
0131 03:02:59.005365 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.005371 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.005448 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.005526 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.006480 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.007619 (+ 1139us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.008829 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.009016 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.012278 (+ 3262us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.012303 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.012307 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.012350 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.014678 (+ 2328us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.015155 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.015161 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.015166 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.015215 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.015499 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.015502 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.015504 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.015678 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.015744 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.016391 (+ 647us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.017060 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260131 03:02:59.017658 13975 heartbeater.cc:344] Connected to a master server at 127.6.22.126:36467
I20260131 03:02:59.018095 13975 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:59.018108 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:58.996914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.124:41045 (local address 127.6.22.65:34943)
0131 03:02:59.005071 (+ 8157us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.005092 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.005246 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.005664 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.005667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.005961 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:59.006334 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.006343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.007764 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.007767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.008693 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.008704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.009166 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.010040 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:59.010058 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:59.012100 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:59.014793 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.014800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.014804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.015041 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.015314 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.015329 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.015333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.015402 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.016121 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:59.016124 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:59.016199 (+ 75us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.017709 (+ 1510us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.017911 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7997,"mutex_wait_us":51,"thread_start_us":125,"threads_started":1}
I20260131 03:02:59.018996 13975 heartbeater.cc:507] Master 127.6.22.126:36467 requested a full tablet report, sending...
I20260131 03:02:59.019030 13974 heartbeater.cc:344] Connected to a master server at 127.6.22.124:41045
I20260131 03:02:59.019145 13974 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:59.019369 13974 heartbeater.cc:507] Master 127.6.22.124:41045 requested a full tablet report, sending...
I20260131 03:02:59.017680 13976 heartbeater.cc:344] Connected to a master server at 127.6.22.125:32917
I20260131 03:02:59.019512 13976 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:59.019730 13976 heartbeater.cc:507] Master 127.6.22.125:32917 requested a full tablet report, sending...
I20260131 03:02:59.020249 12981 ts_manager.cc:194] Re-registered known tserver with Master: dec07f75ac0344b89969f8353b215e19 (127.6.22.65:40815)
I20260131 03:02:59.020880 13777 ts_manager.cc:194] Registered new tserver with Master: dec07f75ac0344b89969f8353b215e19 (127.6.22.65:40815)
I20260131 03:02:59.021159 12981 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:54713
I20260131 03:02:59.023012 13591 ts_manager.cc:194] Registered new tserver with Master: dec07f75ac0344b89969f8353b215e19 (127.6.22.65:40815)
W20260131 03:02:59.126262 13981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:59.126556 13981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:59.126655 13981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:59.130054 13981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:59.130157 13981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:59.130267 13981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:02:59.134469 13981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:44517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=40681
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:36467,127.6.22.125:32917,127.6.22.124:41045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:59.135644 13981 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:59.136641 13981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:59.143136 13989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:59.143224 13981 server_base.cc:1060] running on GCE node
W20260131 03:02:59.143154 13986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:59.143143 13987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:59.143785 13981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:59.144297 13981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:59.145496 13981 hybrid_clock.cc:648] HybridClock initialized: now 1769828579145468 us; error 46 us; skew 500 ppm
Jan 31 03:02:59 dist-test-slave-j368 krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828579, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:59.148095 13981 init.cc:377] Logged in from keytab as oryx/127.6.22.66@KRBTEST.COM (short username oryx)
I20260131 03:02:59.149155 13981 webserver.cc:492] Webserver started at http://127.6.22.66:40681/ using document root <none> and password file <none>
I20260131 03:02:59.149720 13981 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:59.149811 13981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:59.153232 13981 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260131 03:02:59.155058 13996 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:59.156144 13981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:02:59.156293 13981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "256c3d201c82434cb944e16db43d2674"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "f2a0ac2eaa124f40c5cbcf8085856fa9"
server_key_iv: "4576c9f9ee433613e9102ab50ea07251"
server_key_version: "encryptionkey@0"
I20260131 03:02:59.156776 13981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:59.172386 13981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:59.175452 13981 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:59.175670 13981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:59.176865 13981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:59.177841 13981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:59.177927 13981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:59.177996 13981 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:59.178051 13981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:59.188799 13981 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:44517
I20260131 03:02:59.188840 14109 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:44517 every 8 connection(s)
I20260131 03:02:59.190079 13981 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:02:59.193357 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 13981
I20260131 03:02:59.193527 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 13317
I20260131 03:02:59.200320 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:37933
--local_ip_for_outbound_sockets=127.6.22.67
--tserver_master_addrs=127.6.22.126:36467
--webserver_port=43837
--webserver_interface=127.6.22.67
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.22.126:36467,127.6.22.125:32917
--tserver_master_addrs=127.6.22.126:36467,127.6.22.125:32917,127.6.22.124:41045 with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:02:59 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828579, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.124@KRBTEST.COM
Jan 31 03:02:59 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828579, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.125@KRBTEST.COM
Jan 31 03:02:59 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828579, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:02:59.208042 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.191838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:52335 (local address 127.6.22.126:36467)
0131 03:02:59.191989 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.191994 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.192923 (+ 929us) server_negotiation.cc:408] Connection header received
0131 03:02:59.195533 (+ 2610us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.195537 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.195600 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.195650 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.198338 (+ 2688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.199148 (+ 810us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.200205 (+ 1057us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.200434 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.204375 (+ 3941us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.204398 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.204403 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.204434 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.206455 (+ 2021us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.206894 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.206897 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.206899 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.206946 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.207204 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.207207 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.207209 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.207384 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.207464 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.207756 (+ 292us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.207880 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260131 03:02:59.208120 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.192472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:38803 (local address 127.6.22.124:41045)
0131 03:02:59.192622 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.192627 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.193276 (+ 649us) server_negotiation.cc:408] Connection header received
0131 03:02:59.196814 (+ 3538us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.196821 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.196891 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.196951 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.198551 (+ 1600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.199351 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.200242 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.200446 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.203364 (+ 2918us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.203386 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.203390 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.203420 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.205608 (+ 2188us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.206278 (+ 670us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.206284 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.206288 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.206344 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.206741 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.206746 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.206749 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.206964 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.207044 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.207758 (+ 714us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.207898 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260131 03:02:59.208802 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.192103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.125:32917 (local address 127.6.22.66:33829)
0131 03:02:59.193270 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.193285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.195216 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.196015 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.196026 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.196806 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:59.198076 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.198102 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.199313 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.199316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.200205 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.200213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.200903 (+ 690us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.201775 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:59.201797 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:59.203526 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:59.206091 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.206095 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.206098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.206314 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.206588 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.206591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.206593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.206659 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.207090 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:59.207097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:59.207429 (+ 332us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.207665 (+ 236us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.207942 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"mutex_wait_us":826,"spinlock_wait_cycles":15744,"thread_start_us":87,"threads_started":1}
I20260131 03:02:59.208801 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.192330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.124:41045 (local address 127.6.22.66:38803)
0131 03:02:59.193188 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.193199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.195216 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.197106 (+ 1890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.197112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.197526 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:59.198074 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.198099 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.199524 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.199531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.200107 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.200120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.200703 (+ 583us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.201326 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:59.201367 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:59.203179 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:59.205758 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.205766 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.205781 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.206145 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.206482 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.206488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.206492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.206619 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.207127 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:59.207133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:59.207433 (+ 300us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.208274 (+ 841us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.208485 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":48,"spinlock_wait_cycles":12416,"thread_start_us":118,"threads_started":1}
I20260131 03:02:59.208803 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.192085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.66:52335)
0131 03:02:59.192780 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.192820 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.195413 (+ 2593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.196015 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.196026 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.197515 (+ 1489us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:59.198076 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.198102 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.199313 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.199316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.200033 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.200045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.200706 (+ 661us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.201979 (+ 1273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:59.201996 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:59.204202 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:59.206583 (+ 2381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.206586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.206588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.206792 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.207051 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.207057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.207060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.207108 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.207517 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:59.207520 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:59.207587 (+ 67us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.208244 (+ 657us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.208397 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"spinlock_wait_cycles":17792,"thread_start_us":150,"threads_started":1}
I20260131 03:02:59.210019 14111 heartbeater.cc:344] Connected to a master server at 127.6.22.126:36467
I20260131 03:02:59.210022 14110 heartbeater.cc:344] Connected to a master server at 127.6.22.124:41045
I20260131 03:02:59.210294 14110 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:59.210294 14111 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:59.210361 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.192087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:33829 (local address 127.6.22.125:32917)
0131 03:02:59.192256 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.192263 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.193367 (+ 1104us) server_negotiation.cc:408] Connection header received
0131 03:02:59.195460 (+ 2093us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.195464 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.195531 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.195584 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.198335 (+ 2751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.199146 (+ 811us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.200356 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.200557 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.203695 (+ 3138us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.203716 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.203720 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.203753 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.205954 (+ 2201us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.206432 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.206435 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.206438 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.206482 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.206762 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.206766 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.206768 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.206955 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.207042 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.210089 (+ 3047us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.210213 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260131 03:02:59.210901 14111 heartbeater.cc:507] Master 127.6.22.126:36467 requested a full tablet report, sending...
I20260131 03:02:59.210901 14110 heartbeater.cc:507] Master 127.6.22.124:41045 requested a full tablet report, sending...
I20260131 03:02:59.211978 14112 heartbeater.cc:344] Connected to a master server at 127.6.22.125:32917
I20260131 03:02:59.212052 14112 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:59.212036 12981 ts_manager.cc:194] Re-registered known tserver with Master: 256c3d201c82434cb944e16db43d2674 (127.6.22.66:44517)
I20260131 03:02:59.212069 13777 ts_manager.cc:194] Registered new tserver with Master: 256c3d201c82434cb944e16db43d2674 (127.6.22.66:44517)
I20260131 03:02:59.212213 14112 heartbeater.cc:507] Master 127.6.22.125:32917 requested a full tablet report, sending...
I20260131 03:02:59.212749 12981 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:52335
I20260131 03:02:59.212836 13591 ts_manager.cc:194] Registered new tserver with Master: 256c3d201c82434cb944e16db43d2674 (127.6.22.66:44517)
W20260131 03:02:59.311484 14117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:02:59.311764 14117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:02:59.311825 14117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:02:59.315323 14117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:02:59.315380 14117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:02:59.315505 14117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:02:59.319612 14117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:37933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=43837
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:36467,127.6.22.125:32917,127.6.22.124:41045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:02:59.320713 14117 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:02:59.321600 14117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:02:59.327903 14124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:02:59.327893 14123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:59.327947 14117 server_base.cc:1060] running on GCE node
W20260131 03:02:59.327893 14126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:02:59.328434 14117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:02:59.329006 14117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:02:59.330186 14117 hybrid_clock.cc:648] HybridClock initialized: now 1769828579330178 us; error 33 us; skew 500 ppm
Jan 31 03:02:59 dist-test-slave-j368 krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828579, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:02:59.332806 14117 init.cc:377] Logged in from keytab as oryx/127.6.22.67@KRBTEST.COM (short username oryx)
I20260131 03:02:59.333761 14117 webserver.cc:492] Webserver started at http://127.6.22.67:43837/ using document root <none> and password file <none>
I20260131 03:02:59.334342 14117 fs_manager.cc:362] Metadata directory not provided
I20260131 03:02:59.334408 14117 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:02:59.337671 14117 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260131 03:02:59.339429 14133 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:59.340358 14117 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260131 03:02:59.340447 14117 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "35f40d5fb02a49a6828f298950421a75"
format_stamp: "Formatted at 2026-01-31 03:02:55 on dist-test-slave-j368"
server_key: "d6d279540fedabf9370c175c91bbb32f"
server_key_iv: "2d028e7f91e93b35ed910025766546ac"
server_key_version: "encryptionkey@0"
I20260131 03:02:59.340930 14117 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:02:59.362577 14117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:02:59.365021 14117 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:02:59.365301 14117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:02:59.365893 14117 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:02:59.366825 14117 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:02:59.366880 14117 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:59.366950 14117 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:02:59.366990 14117 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:02:59.376507 14117 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:37933
I20260131 03:02:59.376545 14246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:37933 every 8 connection(s)
I20260131 03:02:59.377595 14117 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:02:59.378051 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 14117
Jan 31 03:02:59 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828579, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.125@KRBTEST.COM
Jan 31 03:02:59 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828579, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.124@KRBTEST.COM
Jan 31 03:02:59 dist-test-slave-j368 krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828579, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:02:59.393317 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.380070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:58857 (local address 127.6.22.124:41045)
0131 03:02:59.380181 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.380185 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.380425 (+ 240us) server_negotiation.cc:408] Connection header received
0131 03:02:59.381526 (+ 1101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.381530 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.381583 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.381639 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.383500 (+ 1861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.384211 (+ 711us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.385095 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.385285 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.388835 (+ 3550us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.388853 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.388858 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.388883 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.391034 (+ 2151us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.391662 (+ 628us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.391667 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.391670 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.391724 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.392065 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.392070 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.392073 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.392256 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.392323 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.393026 (+ 703us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.393137 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260131 03:02:59.394081 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.379737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:42633 (local address 127.6.22.125:32917)
0131 03:02:59.379865 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.379870 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.381357 (+ 1487us) server_negotiation.cc:408] Connection header received
0131 03:02:59.381478 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.381481 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.381531 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.381606 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.383494 (+ 1888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.384231 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.385122 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.385320 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.388254 (+ 2934us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.388274 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.388277 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.388304 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.392156 (+ 3852us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.392581 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.392585 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.392586 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.392640 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.392938 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.392944 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.392946 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.393105 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.393172 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.393829 (+ 657us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.393943 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260131 03:02:59.395181 14256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.379558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.124:41045 (local address 127.6.22.67:58857)
0131 03:02:59.380270 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.380308 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.381292 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.381880 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.381894 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.382678 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:59.383335 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.383351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.384360 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.384363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.384978 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.384986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.385494 (+ 508us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.387028 (+ 1534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:59.387044 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:59.388690 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:59.391165 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.391170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.391188 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.391530 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.391834 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.391837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.391839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.391952 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.392413 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:59.392422 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:59.392740 (+ 318us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.393989 (+ 1249us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.394263 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"spinlock_wait_cycles":65664,"thread_start_us":109,"threads_started":1}
I20260131 03:02:59.395177 14255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.379558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.125:32917 (local address 127.6.22.67:42633)
0131 03:02:59.381261 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.381272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.381395 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.381878 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.381891 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.382849 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:59.383338 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.383353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.384367 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.384370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.384994 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.385002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.385493 (+ 491us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.386259 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:59.386306 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:59.388028 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:59.392268 (+ 4240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.392272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.392275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.392479 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.392781 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.392784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.392786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.392837 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.393241 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:59.393244 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:59.393320 (+ 76us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.394400 (+ 1080us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.394516 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1613,"mutex_wait_us":60,"spinlock_wait_cycles":4224,"thread_start_us":120,"threads_started":1}
I20260131 03:02:59.396031 14257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.379558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.67:43343)
0131 03:02:59.381692 (+ 2134us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.381706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.381816 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.382153 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.382158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.382479 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:02:59.383339 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.383353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.384367 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.384383 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.384994 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.385002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.385493 (+ 491us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.386466 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:02:59.386682 (+ 216us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:02:59.389232 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:02:59.393557 (+ 4325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.393560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.393562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.393838 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.394817 (+ 979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:02:59.394821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:02:59.394823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:02:59.394874 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:02:59.395520 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:02:59.395525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:02:59.395608 (+ 83us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.395719 (+ 111us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.395847 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2052,"mutex_wait_us":751,"spinlock_wait_cycles":112512,"thread_start_us":89,"threads_started":1}
I20260131 03:02:59.396364 14249 heartbeater.cc:344] Connected to a master server at 127.6.22.125:32917
I20260131 03:02:59.396376 14247 heartbeater.cc:344] Connected to a master server at 127.6.22.124:41045
I20260131 03:02:59.396621 14249 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:59.396620 14247 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:59.396735 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.379972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:43343 (local address 127.6.22.126:36467)
0131 03:02:59.380205 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.380209 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.381762 (+ 1553us) server_negotiation.cc:408] Connection header received
0131 03:02:59.381913 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.381917 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.381960 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.382013 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.383494 (+ 1481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.384238 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.385123 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.385320 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.389380 (+ 4060us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.389400 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.389403 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.389432 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.392957 (+ 3525us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.393925 (+ 968us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.393929 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.393930 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.393972 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.394973 (+ 1001us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.394977 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.394979 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.395167 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.395255 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.396438 (+ 1183us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.396553 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260131 03:02:59.397284 14247 heartbeater.cc:507] Master 127.6.22.124:41045 requested a full tablet report, sending...
I20260131 03:02:59.397286 14249 heartbeater.cc:507] Master 127.6.22.125:32917 requested a full tablet report, sending...
I20260131 03:02:59.397581 14248 heartbeater.cc:344] Connected to a master server at 127.6.22.126:36467
I20260131 03:02:59.397661 14248 heartbeater.cc:461] Registering TS with master...
I20260131 03:02:59.397814 14248 heartbeater.cc:507] Master 127.6.22.126:36467 requested a full tablet report, sending...
I20260131 03:02:59.398445 13777 ts_manager.cc:194] Registered new tserver with Master: 35f40d5fb02a49a6828f298950421a75 (127.6.22.67:37933)
I20260131 03:02:59.398442 12981 ts_manager.cc:194] Re-registered known tserver with Master: 35f40d5fb02a49a6828f298950421a75 (127.6.22.67:37933)
I20260131 03:02:59.398470 13591 ts_manager.cc:194] Registered new tserver with Master: 35f40d5fb02a49a6828f298950421a75 (127.6.22.67:37933)
I20260131 03:02:59.398979 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.381200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.6.22.125:32917)
0131 03:02:59.381728 (+ 528us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.381733 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.382601 (+ 868us) server_negotiation.cc:408] Connection header received
0131 03:02:59.385898 (+ 3297us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.385903 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.385957 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.386050 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.387585 (+ 1535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.388316 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.390278 (+ 1962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.390519 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.392056 (+ 1537us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.392076 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.392081 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.392116 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.395059 (+ 2943us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.395943 (+ 884us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.395949 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.395953 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.396001 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.397137 (+ 1136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.397140 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.397142 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.397321 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.397429 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.398347 (+ 918us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.398454 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":66,"threads_started":1}
I20260131 03:02:59.399125 12981 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:43343
I20260131 03:02:59.400604 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.380523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35676 (local address 127.6.22.126:36467)
0131 03:02:59.380765 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.380770 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.382562 (+ 1792us) server_negotiation.cc:408] Connection header received
0131 03:02:59.383657 (+ 1095us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.383660 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.383702 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.383754 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.386877 (+ 3123us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.387620 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.388444 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.388635 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.390675 (+ 2040us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.390693 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.390696 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.390727 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.393945 (+ 3218us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.397884 (+ 3939us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.397888 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.397889 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.397933 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.399847 (+ 1914us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.399850 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.399852 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.400000 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.400112 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.400335 (+ 223us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.400455 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":66,"threads_started":1}
I20260131 03:02:59.401108 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.380822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39784 (local address 127.6.22.124:41045)
0131 03:02:59.382138 (+ 1316us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.382143 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.382154 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:02:59.382185 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.382189 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.382226 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.382307 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.384759 (+ 2452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.385462 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.386528 (+ 1066us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.386740 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.389981 (+ 3241us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.389997 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.389999 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.390028 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.394674 (+ 4646us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.397463 (+ 2789us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.397467 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.397469 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.397514 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.397775 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.397779 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.397780 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.399928 (+ 2148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.400039 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.400613 (+ 574us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.400829 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":61,"threads_started":1}
I20260131 03:02:59.403015 12981 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:02:59.405092 12981 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:02:59.419878 14275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.416316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:37933 (local address 127.0.0.1:35266)
0131 03:02:59.416586 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.416601 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.416720 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.417388 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.417391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.417410 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:59.417804 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.417810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.418723 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.418727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.419414 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.419422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.419599 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.419624 (+ 25us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.419689 (+ 65us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.419748 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
I20260131 03:02:59.420473 14276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.416566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.6.22.67:37933)
0131 03:02:59.416883 (+ 317us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.416889 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.416909 (+ 20us) server_negotiation.cc:408] Connection header received
0131 03:02:59.417014 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.417021 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.417234 (+ 213us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.417395 (+ 161us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:59.417903 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.418603 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.419602 (+ 999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.420124 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.420251 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.420306 (+ 55us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.420368 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":140,"threads_started":1}
I20260131 03:02:59.421573 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.417044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:44517 (local address 127.0.0.1:55780)
0131 03:02:59.417719 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.417732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.417809 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.418435 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.418438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.418455 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:59.418719 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.418726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.420208 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.420210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.421122 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.421130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.421271 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.421291 (+ 20us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.421344 (+ 53us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.421409 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":105,"threads_started":1}
I20260131 03:02:59.422122 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.417359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.6.22.66:44517)
0131 03:02:59.418062 (+ 703us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.418068 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.418089 (+ 21us) server_negotiation.cc:408] Connection header received
0131 03:02:59.418169 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.418174 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.418297 (+ 123us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.418443 (+ 146us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:59.418852 (+ 409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.419898 (+ 1046us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.421262 (+ 1364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.421767 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.421861 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.421918 (+ 57us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.421995 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":90,"threads_started":1}
I20260131 03:02:59.423441 14277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.417044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:40815 (local address 127.0.0.1:43518)
0131 03:02:59.417984 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.417996 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.418063 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.419235 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.419238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.419257 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:59.419483 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.419492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.422500 (+ 3008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.422504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.423055 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.423062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.423208 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.423224 (+ 16us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.423273 (+ 49us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.423326 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":227,"thread_start_us":74,"threads_started":1}
I20260131 03:02:59.423221 14181 tablet_service.cc:1505] Processing CreateTablet for tablet da7cb5e0198848fe82c561e63cbe47d6 (DEFAULT_TABLE table=test-table [id=62f40051f1804cdd9246c7246e82288e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:59.424068 14181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da7cb5e0198848fe82c561e63cbe47d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:59.424149 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.417688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.6.22.65:40815)
0131 03:02:59.418854 (+ 1166us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.418859 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.418881 (+ 22us) server_negotiation.cc:408] Connection header received
0131 03:02:59.418960 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.418965 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.419098 (+ 133us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.419224 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:59.419643 (+ 419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.422367 (+ 2724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.423203 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.423795 (+ 592us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.423865 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.423911 (+ 46us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.424008 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1040,"thread_start_us":102,"threads_started":1}
I20260131 03:02:59.426759 13908 tablet_service.cc:1505] Processing CreateTablet for tablet da7cb5e0198848fe82c561e63cbe47d6 (DEFAULT_TABLE table=test-table [id=62f40051f1804cdd9246c7246e82288e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:59.426859 14044 tablet_service.cc:1505] Processing CreateTablet for tablet da7cb5e0198848fe82c561e63cbe47d6 (DEFAULT_TABLE table=test-table [id=62f40051f1804cdd9246c7246e82288e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:02:59.427672 14044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da7cb5e0198848fe82c561e63cbe47d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:59.427672 13908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da7cb5e0198848fe82c561e63cbe47d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:02:59.430446 14281 tablet_bootstrap.cc:492] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75: Bootstrap starting.
I20260131 03:02:59.432384 14281 tablet_bootstrap.cc:654] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:59.433225 14281 log.cc:826] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:59.434418 14283 tablet_bootstrap.cc:492] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674: Bootstrap starting.
I20260131 03:02:59.434850 14284 tablet_bootstrap.cc:492] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19: Bootstrap starting.
I20260131 03:02:59.436239 14281 tablet_bootstrap.cc:492] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75: No bootstrap required, opened a new log
I20260131 03:02:59.436460 14281 ts_tablet_manager.cc:1403] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260131 03:02:59.436496 14283 tablet_bootstrap.cc:654] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:59.437108 14284 tablet_bootstrap.cc:654] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19: Neither blocks nor log segments found. Creating new log.
I20260131 03:02:59.437309 14283 log.cc:826] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:59.437856 14284 log.cc:826] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19: Log is configured to *not* fsync() on all Append() calls
I20260131 03:02:59.439036 14283 tablet_bootstrap.cc:492] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674: No bootstrap required, opened a new log
I20260131 03:02:59.439245 14283 ts_tablet_manager.cc:1403] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260131 03:02:59.439576 14284 tablet_bootstrap.cc:492] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19: No bootstrap required, opened a new log
I20260131 03:02:59.439523 14281 raft_consensus.cc:359] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } } peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } }
I20260131 03:02:59.439736 14284 ts_tablet_manager.cc:1403] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260131 03:02:59.439752 14281 raft_consensus.cc:385] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:59.439803 14281 raft_consensus.cc:740] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f40d5fb02a49a6828f298950421a75, State: Initialized, Role: FOLLOWER
I20260131 03:02:59.440269 14281 consensus_queue.cc:260] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } } peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } }
I20260131 03:02:59.441113 14281 ts_tablet_manager.cc:1434] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260131 03:02:59.441294 14248 heartbeater.cc:499] Master 127.6.22.126:36467 was elected leader, sending a full tablet report...
I20260131 03:02:59.442584 14284 raft_consensus.cc:359] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } } peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } }
I20260131 03:02:59.442782 14284 raft_consensus.cc:385] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:59.442826 14284 raft_consensus.cc:740] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec07f75ac0344b89969f8353b215e19, State: Initialized, Role: FOLLOWER
I20260131 03:02:59.443198 14284 consensus_queue.cc:260] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } } peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } }
I20260131 03:02:59.443197 14283 raft_consensus.cc:359] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } } peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } }
I20260131 03:02:59.443354 14283 raft_consensus.cc:385] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:02:59.443400 14283 raft_consensus.cc:740] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256c3d201c82434cb944e16db43d2674, State: Initialized, Role: FOLLOWER
I20260131 03:02:59.443792 14283 consensus_queue.cc:260] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } } peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } }
I20260131 03:02:59.443969 14284 ts_tablet_manager.cc:1434] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260131 03:02:59.444289 13975 heartbeater.cc:499] Master 127.6.22.126:36467 was elected leader, sending a full tablet report...
I20260131 03:02:59.445470 14283 ts_tablet_manager.cc:1434] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260131 03:02:59.445820 14111 heartbeater.cc:499] Master 127.6.22.126:36467 was elected leader, sending a full tablet report...
W20260131 03:02:59.506292 13977 tablet.cc:2378] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:02:59.628865 14250 tablet.cc:2378] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:02:59.691383 14113 tablet.cc:2378] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:02:59.743062 14287 raft_consensus.cc:493] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:02:59.743261 14287 raft_consensus.cc:515] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } } peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } }
I20260131 03:02:59.744354 14287 leader_election.cc:290] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec07f75ac0344b89969f8353b215e19 (127.6.22.65:40815), 256c3d201c82434cb944e16db43d2674 (127.6.22.66:44517)
I20260131 03:02:59.748023 14255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.744753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:44517 (local address 127.6.22.67:50779)
0131 03:02:59.744981 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.744998 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.745105 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.745434 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.745438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.745464 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:59.745748 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.745755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.746878 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.746882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.747472 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.747480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.747633 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.747657 (+ 24us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.747713 (+ 56us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.747777 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"spinlock_wait_cycles":19840}
I20260131 03:02:59.748021 14257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.744753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:40815 (local address 127.6.22.67:44241)
0131 03:02:59.744981 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0131 03:02:59.744998 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:02:59.745105 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:02:59.745434 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:02:59.745447 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:02:59.745469 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:02:59.745748 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.745755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.746838 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.746844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:02:59.747504 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:02:59.747512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.747656 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.747675 (+ 19us) client_negotiation.cc:769] Sending connection context
0131 03:02:59.747728 (+ 53us) client_negotiation.cc:240] Negotiation successful
0131 03:02:59.747784 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":58,"spinlock_wait_cycles":13312}
I20260131 03:02:59.748411 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.744903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:50779 (local address 127.6.22.66:44517)
0131 03:02:59.745034 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.745040 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.745058 (+ 18us) server_negotiation.cc:408] Connection header received
0131 03:02:59.745245 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.745249 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.745302 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.745401 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:59.745923 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.746697 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.747608 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.748136 (+ 528us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.748181 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.748242 (+ 61us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.748308 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260131 03:02:59.748432 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.744903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:44241 (local address 127.6.22.65:40815)
0131 03:02:59.745034 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.745040 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.745058 (+ 18us) server_negotiation.cc:408] Connection header received
0131 03:02:59.745245 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.745249 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.745302 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.745401 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:02:59.745923 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.746697 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.747640 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.748140 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.748183 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.748239 (+ 56us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.748303 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260131 03:02:59.749013 13928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da7cb5e0198848fe82c561e63cbe47d6" candidate_uuid: "35f40d5fb02a49a6828f298950421a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec07f75ac0344b89969f8353b215e19" is_pre_election: true
I20260131 03:02:59.749015 14062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da7cb5e0198848fe82c561e63cbe47d6" candidate_uuid: "35f40d5fb02a49a6828f298950421a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "256c3d201c82434cb944e16db43d2674" is_pre_election: true
I20260131 03:02:59.749290 14062 raft_consensus.cc:2468] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f40d5fb02a49a6828f298950421a75 in term 0.
I20260131 03:02:59.749297 13928 raft_consensus.cc:2468] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f40d5fb02a49a6828f298950421a75 in term 0.
I20260131 03:02:59.749773 14137 leader_election.cc:304] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 256c3d201c82434cb944e16db43d2674, 35f40d5fb02a49a6828f298950421a75; no voters:
I20260131 03:02:59.750066 14287 raft_consensus.cc:2804] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:02:59.750145 14287 raft_consensus.cc:493] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:02:59.750190 14287 raft_consensus.cc:3060] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:59.751379 14287 raft_consensus.cc:515] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } } peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } }
I20260131 03:02:59.751775 14287 leader_election.cc:290] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [CANDIDATE]: Term 1 election: Requested vote from peers dec07f75ac0344b89969f8353b215e19 (127.6.22.65:40815), 256c3d201c82434cb944e16db43d2674 (127.6.22.66:44517)
I20260131 03:02:59.752111 13928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da7cb5e0198848fe82c561e63cbe47d6" candidate_uuid: "35f40d5fb02a49a6828f298950421a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec07f75ac0344b89969f8353b215e19"
I20260131 03:02:59.752185 14062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da7cb5e0198848fe82c561e63cbe47d6" candidate_uuid: "35f40d5fb02a49a6828f298950421a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "256c3d201c82434cb944e16db43d2674"
I20260131 03:02:59.752250 13928 raft_consensus.cc:3060] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:59.752336 14062 raft_consensus.cc:3060] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:02:59.753289 14062 raft_consensus.cc:2468] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f40d5fb02a49a6828f298950421a75 in term 1.
I20260131 03:02:59.753290 13928 raft_consensus.cc:2468] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f40d5fb02a49a6828f298950421a75 in term 1.
I20260131 03:02:59.753624 14137 leader_election.cc:304] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 256c3d201c82434cb944e16db43d2674, 35f40d5fb02a49a6828f298950421a75; no voters:
I20260131 03:02:59.753844 14287 raft_consensus.cc:2804] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:02:59.754092 14287 raft_consensus.cc:697] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 1 LEADER]: Becoming Leader. State: Replica: 35f40d5fb02a49a6828f298950421a75, State: Running, Role: LEADER
I20260131 03:02:59.754462 14287 consensus_queue.cc:237] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } } peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } }
I20260131 03:02:59.758328 12979 catalog_manager.cc:5654] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f40d5fb02a49a6828f298950421a75 (127.6.22.67). New cstate: current_term: 1 leader_uuid: "35f40d5fb02a49a6828f298950421a75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260131 03:02:59.790549 14276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.787275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.6.22.67:37933)
0131 03:02:59.787417 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.787420 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.787434 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:02:59.787514 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.787517 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.787572 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.787655 (+ 83us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:02:59.787985 (+ 330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.788468 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.789244 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.789403 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.789515 (+ 112us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:02:59.789974 (+ 459us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:02:59.790078 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.790339 (+ 261us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.790410 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260131 03:02:59.797364 14062 raft_consensus.cc:1275] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674 [term 1 FOLLOWER]: Refusing update from remote peer 35f40d5fb02a49a6828f298950421a75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:59.797737 13928 raft_consensus.cc:1275] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19 [term 1 FOLLOWER]: Refusing update from remote peer 35f40d5fb02a49a6828f298950421a75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:02:59.798122 14287 consensus_queue.cc:1048] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:59.798365 14290 consensus_queue.cc:1048] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:02:59.806809 14293 mvcc.cc:204] Tried to move back new op lower bound from 7249217862842490880 to 7249217862677286912. Current Snapshot: MvccSnapshot[applied={T|T < 7249217862842490880}]
I20260131 03:02:59.806967 14294 mvcc.cc:204] Tried to move back new op lower bound from 7249217862842490880 to 7249217862677286912. Current Snapshot: MvccSnapshot[applied={T|T < 7249217862842490880}]
I20260131 03:02:59.808835 14295 mvcc.cc:204] Tried to move back new op lower bound from 7249217862842490880 to 7249217862677286912. Current Snapshot: MvccSnapshot[applied={T|T < 7249217862842490880}]
I20260131 03:02:59.818799 12979 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:59.818962 12979 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:02:59.825125 12979 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 2abf73bb02194843aaeb35f43d12cf52: Sending DeleteTablet for 3 replicas of tablet da7cb5e0198848fe82c561e63cbe47d6
I20260131 03:02:59.825922 14181 tablet_service.cc:1552] Processing DeleteTablet for tablet da7cb5e0198848fe82c561e63cbe47d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:59 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:35266
I20260131 03:02:59.825994 13908 tablet_service.cc:1552] Processing DeleteTablet for tablet da7cb5e0198848fe82c561e63cbe47d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:59 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:43518
I20260131 03:02:59.826555 14302 tablet_replica.cc:333] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19: stopping tablet replica
I20260131 03:02:59.826579 14303 tablet_replica.cc:333] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75: stopping tablet replica
I20260131 03:02:59.826856 14303 raft_consensus.cc:2243] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:02:59.826858 14302 raft_consensus.cc:2243] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:59.827291 14303 raft_consensus.cc:2272] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:59.827545 14302 raft_consensus.cc:2272] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:59.828678 14303 ts_tablet_manager.cc:1916] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:59.828961 14302 ts_tablet_manager.cc:1916] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:59.829564 14044 tablet_service.cc:1552] Processing DeleteTablet for tablet da7cb5e0198848fe82c561e63cbe47d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:02:59 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:55780
I20260131 03:02:59.830183 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 12949
I20260131 03:02:59.831842 14303 ts_tablet_manager.cc:1929] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:02:59.831990 14303 log.cc:1199] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/da7cb5e0198848fe82c561e63cbe47d6
I20260131 03:02:59.832401 14303 ts_tablet_manager.cc:1950] T da7cb5e0198848fe82c561e63cbe47d6 P 35f40d5fb02a49a6828f298950421a75: Deleting consensus metadata
I20260131 03:02:59.832481 14302 ts_tablet_manager.cc:1929] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:02:59.832573 14302 log.cc:1199] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-0/wal/wals/da7cb5e0198848fe82c561e63cbe47d6
I20260131 03:02:59.833105 14302 ts_tablet_manager.cc:1950] T da7cb5e0198848fe82c561e63cbe47d6 P dec07f75ac0344b89969f8353b215e19: Deleting consensus metadata
W20260131 03:02:59.837267 13864 connection.cc:537] server connection from 127.0.0.1:43518 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260131 03:02:59.837297 14137 connection.cc:537] server connection from 127.0.0.1:35266 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260131 03:02:59.837641 14304 tablet_replica.cc:333] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674: stopping tablet replica
I20260131 03:02:59.837939 14304 raft_consensus.cc:2243] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:02:59.838299 14304 raft_consensus.cc:2272] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:02:59.839839 14304 ts_tablet_manager.cc:1916] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:02:59.842576 14304 ts_tablet_manager.cc:1929] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:02:59.842736 14304 log.cc:1199] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/da7cb5e0198848fe82c561e63cbe47d6
I20260131 03:02:59.843125 14304 ts_tablet_manager.cc:1950] T da7cb5e0198848fe82c561e63cbe47d6 P 256c3d201c82434cb944e16db43d2674: Deleting consensus metadata
W20260131 03:02:59.844228 13998 connection.cc:441] server connection from 127.0.0.1:55780 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55780 (request call id 1) could send its response
I20260131 03:02:59.851706 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.842713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.6.22.125:32917)
0131 03:02:59.842857 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.842865 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.843087 (+ 222us) server_negotiation.cc:408] Connection header received
0131 03:02:59.843289 (+ 202us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.843294 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.843352 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.843417 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.844984 (+ 1567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.845797 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.846591 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.846755 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.848156 (+ 1401us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.848175 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.848178 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.848211 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.849888 (+ 1677us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.850356 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.850360 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.850362 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.850415 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.850736 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.850740 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.850744 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.850928 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.851068 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.851338 (+ 270us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.851522 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:02:59.851981 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:02:59.842713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48852 (local address 127.6.22.124:41045)
0131 03:02:59.842843 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0131 03:02:59.842849 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:02:59.843196 (+ 347us) server_negotiation.cc:408] Connection header received
0131 03:02:59.843352 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:02:59.843356 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:02:59.843409 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:02:59.843457 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:02:59.845095 (+ 1638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.845604 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:02:59.846444 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:02:59.846658 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:02:59.848006 (+ 1348us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:02:59.848026 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:02:59.848030 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:02:59.848058 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:02:59.850096 (+ 2038us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.850507 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.850510 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.850513 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.850558 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:02:59.850805 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:02:59.850809 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:02:59.850811 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:02:59.850986 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:02:59.851108 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0131 03:02:59.851691 (+ 583us) server_negotiation.cc:300] Negotiation successful
0131 03:02:59.851820 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260131 03:03:00.811290 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:00.810803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.66:48019)
0131 03:03:00.811077 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:00.811159 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
W20260131 03:03:00.811553 14111 heartbeater.cc:646] Failed to heartbeat to 127.6.22.126:36467 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
I20260131 03:03:00.812170 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:00.811331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.65:56755)
0131 03:03:00.812005 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:00.812075 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":59,"threads_started":1}
I20260131 03:03:00.812546 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:00.812094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.67:34709)
0131 03:03:00.812387 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:00.812453 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":56,"threads_started":1}
W20260131 03:03:00.812547 13975 heartbeater.cc:646] Failed to heartbeat to 127.6.22.126:36467 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
W20260131 03:03:00.812837 14248 heartbeater.cc:646] Failed to heartbeat to 127.6.22.126:36467 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
I20260131 03:03:01.324759 14315 raft_consensus.cc:493] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2abf73bb02194843aaeb35f43d12cf52)
I20260131 03:03:01.324946 14315 raft_consensus.cc:515] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } }
I20260131 03:03:01.325176 14316 raft_consensus.cc:493] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2abf73bb02194843aaeb35f43d12cf52)
I20260131 03:03:01.325346 14316 raft_consensus.cc:515] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } }
I20260131 03:03:01.326103 14315 leader_election.cc:290] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467), 54efaa3b58ae4b939212378c1ea9591e (127.6.22.124:41045)
I20260131 03:03:01.326582 13787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4958e667bad4512a95edb0f583dee0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "54efaa3b58ae4b939212378c1ea9591e" is_pre_election: true
I20260131 03:03:01.326699 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:01.326246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60518)
0131 03:03:01.326517 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:01.326596 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":59,"threads_started":1}
I20260131 03:03:01.326747 13787 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4958e667bad4512a95edb0f583dee0a in term 1.
W20260131 03:03:01.326823 13578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
I20260131 03:03:01.326723 14316 leader_election.cc:290] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467), f4958e667bad4512a95edb0f583dee0a (127.6.22.125:32917)
I20260131 03:03:01.326711 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:01.326316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60532)
0131 03:03:01.326540 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:01.326602 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260131 03:03:01.326984 13601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54efaa3b58ae4b939212378c1ea9591e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f4958e667bad4512a95edb0f583dee0a" is_pre_election: true
W20260131 03:03:01.327059 13764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
I20260131 03:03:01.327132 13601 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54efaa3b58ae4b939212378c1ea9591e in term 1.
I20260131 03:03:01.327159 13577 leader_election.cc:304] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54efaa3b58ae4b939212378c1ea9591e, f4958e667bad4512a95edb0f583dee0a; no voters:
I20260131 03:03:01.327376 14315 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260131 03:03:01.327425 14315 raft_consensus.cc:493] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2abf73bb02194843aaeb35f43d12cf52)
I20260131 03:03:01.327500 14315 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 1 FOLLOWER]: Advancing to term 2
I20260131 03:03:01.327469 13762 leader_election.cc:304] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54efaa3b58ae4b939212378c1ea9591e, f4958e667bad4512a95edb0f583dee0a; no voters:
I20260131 03:03:01.327672 14316 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260131 03:03:01.327734 14316 raft_consensus.cc:493] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2abf73bb02194843aaeb35f43d12cf52)
I20260131 03:03:01.327790 14316 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 1 FOLLOWER]: Advancing to term 2
I20260131 03:03:01.328214 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:01.327962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60556)
0131 03:03:01.328106 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:01.328156 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260131 03:03:01.328382 13578 leader_election.cc:336] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467): Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
I20260131 03:03:01.328481 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:01.327613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60548)
0131 03:03:01.328386 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:01.328429 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":707}
W20260131 03:03:01.328603 13764 leader_election.cc:336] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467): Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
I20260131 03:03:01.328956 14315 raft_consensus.cc:515] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } }
I20260131 03:03:01.328987 14316 raft_consensus.cc:515] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } }
I20260131 03:03:01.329483 14316 leader_election.cc:290] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [CANDIDATE]: Term 2 election: Requested vote from peers 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467), f4958e667bad4512a95edb0f583dee0a (127.6.22.125:32917)
I20260131 03:03:01.329613 13787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4958e667bad4512a95edb0f583dee0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "54efaa3b58ae4b939212378c1ea9591e"
I20260131 03:03:01.329676 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:01.329416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60562)
0131 03:03:01.329569 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:01.329615 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260131 03:03:01.329772 13787 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f4958e667bad4512a95edb0f583dee0a in current term 2: Already voted for candidate 54efaa3b58ae4b939212378c1ea9591e in this term.
I20260131 03:03:01.329788 13601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54efaa3b58ae4b939212378c1ea9591e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f4958e667bad4512a95edb0f583dee0a"
I20260131 03:03:01.329916 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:01.329519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60578)
0131 03:03:01.329810 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:01.329858 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260131 03:03:01.329921 13601 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 54efaa3b58ae4b939212378c1ea9591e in current term 2: Already voted for candidate f4958e667bad4512a95edb0f583dee0a in this term.
I20260131 03:03:01.330057 14315 leader_election.cc:290] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [CANDIDATE]: Term 2 election: Requested vote from peers 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467), 54efaa3b58ae4b939212378c1ea9591e (127.6.22.124:41045)
I20260131 03:03:01.330511 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:01.330264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60592)
0131 03:03:01.330374 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:01.330419 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260131 03:03:01.330638 13578 leader_election.cc:336] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467): Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
I20260131 03:03:01.330713 13578 leader_election.cc:304] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4958e667bad4512a95edb0f583dee0a; no voters: 2abf73bb02194843aaeb35f43d12cf52, 54efaa3b58ae4b939212378c1ea9591e
I20260131 03:03:01.330871 14315 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260131 03:03:01.331290 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:01.331073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60606)
0131 03:03:01.331179 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:01.331229 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260131 03:03:01.331496 13764 leader_election.cc:336] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467): Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
I20260131 03:03:01.331579 13764 leader_election.cc:304] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54efaa3b58ae4b939212378c1ea9591e; no voters: 2abf73bb02194843aaeb35f43d12cf52, f4958e667bad4512a95edb0f583dee0a
I20260131 03:03:01.331849 14316 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260131 03:03:01.812583 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:01.812111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.66:34169)
0131 03:03:01.812400 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:01.812462 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
I20260131 03:03:01.813346 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:01.812951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.65:42149)
0131 03:03:01.813216 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:01.813273 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":91,"threads_started":1}
I20260131 03:03:01.813742 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:01.813338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.67:50449)
0131 03:03:01.813625 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:01.813669 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":87,"threads_started":1}
I20260131 03:03:02.814472 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.813905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.65:58969)
0131 03:03:02.814290 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:02.814349 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":62,"threads_started":1}
I20260131 03:03:02.814813 14324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.813413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.66:57187)
0131 03:03:02.814551 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:02.814719 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1057,"thread_start_us":78,"threads_started":1}
I20260131 03:03:02.814908 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.814331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.67:45977)
0131 03:03:02.814735 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:02.814787 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":73,"threads_started":1}
I20260131 03:03:02.934859 14327 raft_consensus.cc:493] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:03:02.935055 14327 raft_consensus.cc:515] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } }
I20260131 03:03:02.935527 14327 leader_election.cc:290] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467), 54efaa3b58ae4b939212378c1ea9591e (127.6.22.124:41045)
I20260131 03:03:02.936094 13787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4958e667bad4512a95edb0f583dee0a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "54efaa3b58ae4b939212378c1ea9591e" is_pre_election: true
I20260131 03:03:02.936285 13787 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4958e667bad4512a95edb0f583dee0a in term 2.
I20260131 03:03:02.936304 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.935871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60844)
0131 03:03:02.936116 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:02.936167 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":102,"threads_started":1}
I20260131 03:03:02.936642 13577 leader_election.cc:304] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54efaa3b58ae4b939212378c1ea9591e, f4958e667bad4512a95edb0f583dee0a; no voters:
I20260131 03:03:02.937103 14327 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260131 03:03:02.937171 14327 raft_consensus.cc:493] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:03:02.937211 14327 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 2 FOLLOWER]: Advancing to term 3
I20260131 03:03:02.937216 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.936977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60860)
0131 03:03:02.937095 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:02.937144 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260131 03:03:02.937330 13578 leader_election.cc:336] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467): Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
I20260131 03:03:02.938259 14327 raft_consensus.cc:515] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } }
I20260131 03:03:02.938678 14327 leader_election.cc:290] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [CANDIDATE]: Term 3 election: Requested vote from peers 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467), 54efaa3b58ae4b939212378c1ea9591e (127.6.22.124:41045)
I20260131 03:03:02.938997 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.938765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60864)
0131 03:03:02.938860 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:02.938911 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260131 03:03:02.939210 13787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4958e667bad4512a95edb0f583dee0a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "54efaa3b58ae4b939212378c1ea9591e"
I20260131 03:03:02.939349 13787 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 2 FOLLOWER]: Advancing to term 3
I20260131 03:03:02.939685 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.939500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60874)
0131 03:03:02.939601 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:02.939640 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260131 03:03:02.939873 13578 leader_election.cc:336] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467): Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
I20260131 03:03:02.940447 13787 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4958e667bad4512a95edb0f583dee0a in term 3.
I20260131 03:03:02.940797 13577 leader_election.cc:304] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54efaa3b58ae4b939212378c1ea9591e, f4958e667bad4512a95edb0f583dee0a; no voters: 2abf73bb02194843aaeb35f43d12cf52
I20260131 03:03:02.940968 14327 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 3 FOLLOWER]: Leader election won for term 3
I20260131 03:03:02.941138 14327 raft_consensus.cc:697] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [term 3 LEADER]: Becoming Leader. State: Replica: f4958e667bad4512a95edb0f583dee0a, State: Running, Role: LEADER
I20260131 03:03:02.941421 14327 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } }
I20260131 03:03:02.942515 14330 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4958e667bad4512a95edb0f583dee0a. Latest consensus state: current_term: 3 leader_uuid: "f4958e667bad4512a95edb0f583dee0a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } } }
I20260131 03:03:02.942639 14330 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:02.942941 14332 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:03:02.945118 14332 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:03:02.945531 14332 catalog_manager.cc:1269] Loaded cluster ID: 36b79362917a44fb8ac3b28dff189dc6
I20260131 03:03:02.945577 14332 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:03:02.946142 14332 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:03:02.946552 14332 catalog_manager.cc:6038] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: Loaded TSK: 0
I20260131 03:03:02.947244 14332 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:03:02.975155 13590 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:03:02.977540 13590 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:03:02.980573 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.980293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60890)
0131 03:03:02.980427 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:02.980489 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260131 03:03:02.980752 13787 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [term 3 FOLLOWER]: Refusing update from remote peer f4958e667bad4512a95edb0f583dee0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260131 03:03:02.981422 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.981151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.0.0.1:60902)
0131 03:03:02.981295 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:02.981348 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260131 03:03:02.981421 14327 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260131 03:03:02.981627 13578 consensus_peers.cc:597] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a -> Peer 2abf73bb02194843aaeb35f43d12cf52 (127.6.22.126:36467): Couldn't send request to peer 2abf73bb02194843aaeb35f43d12cf52. Status: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260131 03:03:02.982921 14334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4958e667bad4512a95edb0f583dee0a. Latest consensus state: current_term: 3 leader_uuid: "f4958e667bad4512a95edb0f583dee0a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } } }
I20260131 03:03:02.983043 14334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:03:02.984115 14330 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f4958e667bad4512a95edb0f583dee0a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } } }
I20260131 03:03:02.984226 14330 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:02.984439 14335 mvcc.cc:204] Tried to move back new op lower bound from 7249217875883663360 to 7249217875730575360. Current Snapshot: MvccSnapshot[applied={T|T < 7249217875883663360}]
I20260131 03:03:02.984591 14334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f4958e667bad4512a95edb0f583dee0a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2abf73bb02194843aaeb35f43d12cf52" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 36467 } } peers { permanent_uuid: "f4958e667bad4512a95edb0f583dee0a" member_type: VOTER last_known_addr { host: "127.6.22.125" port: 32917 } attrs { promote: false } } peers { permanent_uuid: "54efaa3b58ae4b939212378c1ea9591e" member_type: VOTER last_known_addr { host: "127.6.22.124" port: 41045 } attrs { promote: false } } }
I20260131 03:03:02.984714 14334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54efaa3b58ae4b939212378c1ea9591e [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:03:02.994612 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.991009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:40815 (local address 127.0.0.1:44650)
0131 03:03:02.991364 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:02.991382 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:02.991472 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:02.991984 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:02.991988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:02.992006 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:02.992248 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:02.992255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:02.993323 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:02.993326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:02.994258 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:02.994269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:02.994342 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:02.994362 (+ 20us) client_negotiation.cc:769] Sending connection context
0131 03:03:02.994397 (+ 35us) client_negotiation.cc:240] Negotiation successful
0131 03:03:02.994459 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274}
I20260131 03:03:02.995329 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.991756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:37933 (local address 127.0.0.1:32838)
0131 03:03:02.992355 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:02.992369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:02.992482 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:02.992800 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:02.992803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:02.992822 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:02.993111 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:02.993120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:02.994154 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:02.994157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:02.994953 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:02.994962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:02.995087 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:02.995102 (+ 15us) client_negotiation.cc:769] Sending connection context
0131 03:03:02.995151 (+ 49us) client_negotiation.cc:240] Negotiation successful
0131 03:03:02.995210 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":85,"threads_started":1}
I20260131 03:03:02.995466 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.991426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:44517 (local address 127.0.0.1:49136)
0131 03:03:02.991706 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:02.991721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:02.991800 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:02.993003 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:02.993006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:02.993024 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:02.993254 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:02.993264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:02.994283 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:02.994287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:02.995134 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:02.995143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:02.995242 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:02.995257 (+ 15us) client_negotiation.cc:769] Sending connection context
0131 03:03:02.995304 (+ 47us) client_negotiation.cc:240] Negotiation successful
0131 03:03:02.995356 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
I20260131 03:03:02.995824 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.992094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.6.22.67:37933)
0131 03:03:02.992356 (+ 262us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:02.992362 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:02.992442 (+ 80us) server_negotiation.cc:408] Connection header received
0131 03:03:02.992599 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:02.992602 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:02.992684 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:02.992771 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:02.993252 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:02.994026 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:02.995075 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:02.995590 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:02.995635 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:02.995678 (+ 43us) server_negotiation.cc:300] Negotiation successful
0131 03:03:02.995729 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
I20260131 03:03:02.996402 14342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.991529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.6.22.66:44517)
0131 03:03:02.992761 (+ 1232us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:02.992767 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:02.992780 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:03:02.992830 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:02.992833 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:02.992881 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:02.992994 (+ 113us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:02.993396 (+ 402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:02.994166 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:02.995665 (+ 1499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:02.996117 (+ 452us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:02.996158 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:02.996210 (+ 52us) server_negotiation.cc:300] Negotiation successful
0131 03:03:02.996268 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1113,"thread_start_us":60,"threads_started":1}
I20260131 03:03:02.996620 14181 tablet_service.cc:1505] Processing CreateTablet for tablet dadfa1fe33c848288f5ee26f567faf82 (DEFAULT_TABLE table=test-table [id=29fb508aae83456fa0a03e5c2ec10392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:02.996958 14181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dadfa1fe33c848288f5ee26f567faf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:02.998155 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:02.991130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.6.22.65:40815)
0131 03:03:02.991703 (+ 573us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:02.991708 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:02.991725 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:03:02.991784 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:02.991788 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:02.991858 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:02.992342 (+ 484us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:02.992426 (+ 84us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:02.993210 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:02.997420 (+ 4210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:02.997916 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:02.997960 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:02.998005 (+ 45us) server_negotiation.cc:300] Negotiation successful
0131 03:03:02.998065 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20260131 03:03:02.998662 13908 tablet_service.cc:1505] Processing CreateTablet for tablet dadfa1fe33c848288f5ee26f567faf82 (DEFAULT_TABLE table=test-table [id=29fb508aae83456fa0a03e5c2ec10392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:02.998955 13908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dadfa1fe33c848288f5ee26f567faf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:03.000011 14345 tablet_bootstrap.cc:492] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75: Bootstrap starting.
I20260131 03:03:03.000530 14044 tablet_service.cc:1505] Processing CreateTablet for tablet dadfa1fe33c848288f5ee26f567faf82 (DEFAULT_TABLE table=test-table [id=29fb508aae83456fa0a03e5c2ec10392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:03.001127 14044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dadfa1fe33c848288f5ee26f567faf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:03.002110 14345 tablet_bootstrap.cc:654] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:03.003858 14345 tablet_bootstrap.cc:492] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75: No bootstrap required, opened a new log
I20260131 03:03:03.003939 14345 ts_tablet_manager.cc:1403] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260131 03:03:03.004441 14345 raft_consensus.cc:359] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } } peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } }
I20260131 03:03:03.004408 14348 tablet_bootstrap.cc:492] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674: Bootstrap starting.
I20260131 03:03:03.004607 14346 tablet_bootstrap.cc:492] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19: Bootstrap starting.
I20260131 03:03:03.004755 14345 raft_consensus.cc:385] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:03.004812 14345 raft_consensus.cc:740] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f40d5fb02a49a6828f298950421a75, State: Initialized, Role: FOLLOWER
I20260131 03:03:03.004984 14345 consensus_queue.cc:260] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } } peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } }
I20260131 03:03:03.005398 14345 ts_tablet_manager.cc:1434] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260131 03:03:03.006114 14348 tablet_bootstrap.cc:654] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:03.006443 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.006000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.67:60003)
0131 03:03:03.006282 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.006348 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260131 03:03:03.007292 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.007023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.67:58657)
0131 03:03:03.007185 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.007233 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260131 03:03:03.008317 14346 tablet_bootstrap.cc:654] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:03.008344 14249 heartbeater.cc:499] Master 127.6.22.125:32917 was elected leader, sending a full tablet report...
I20260131 03:03:03.009654 14348 tablet_bootstrap.cc:492] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674: No bootstrap required, opened a new log
I20260131 03:03:03.009822 14348 ts_tablet_manager.cc:1403] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260131 03:03:03.010298 14348 raft_consensus.cc:359] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } } peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } }
I20260131 03:03:03.010428 14348 raft_consensus.cc:385] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:03.010474 14348 raft_consensus.cc:740] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256c3d201c82434cb944e16db43d2674, State: Initialized, Role: FOLLOWER
I20260131 03:03:03.010638 14348 consensus_queue.cc:260] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } } peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } }
I20260131 03:03:03.011020 14348 ts_tablet_manager.cc:1434] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260131 03:03:03.011766 14324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.011469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.66:52661)
0131 03:03:03.011615 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.011677 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260131 03:03:03.012727 14324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.012477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.66:42383)
0131 03:03:03.012584 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.012631 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260131 03:03:03.012928 14112 heartbeater.cc:499] Master 127.6.22.125:32917 was elected leader, sending a full tablet report...
I20260131 03:03:03.013334 14346 tablet_bootstrap.cc:492] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19: No bootstrap required, opened a new log
I20260131 03:03:03.013494 14346 ts_tablet_manager.cc:1403] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20260131 03:03:03.013955 14346 raft_consensus.cc:359] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } } peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } }
I20260131 03:03:03.014083 14346 raft_consensus.cc:385] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:03.014127 14346 raft_consensus.cc:740] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec07f75ac0344b89969f8353b215e19, State: Initialized, Role: FOLLOWER
I20260131 03:03:03.014310 14346 consensus_queue.cc:260] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } } peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } }
I20260131 03:03:03.014703 14346 ts_tablet_manager.cc:1434] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260131 03:03:03.015352 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.015000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.65:34373)
0131 03:03:03.015223 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.015280 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260131 03:03:03.016242 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.015965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.65:42979)
0131 03:03:03.016115 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.016167 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260131 03:03:03.016458 13976 heartbeater.cc:499] Master 127.6.22.125:32917 was elected leader, sending a full tablet report...
I20260131 03:03:03.305536 14352 raft_consensus.cc:493] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:03:03.305759 14352 raft_consensus.cc:515] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } } peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } }
I20260131 03:03:03.307274 14352 leader_election.cc:290] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec07f75ac0344b89969f8353b215e19 (127.6.22.65:40815), 35f40d5fb02a49a6828f298950421a75 (127.6.22.67:37933)
I20260131 03:03:03.310357 14324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.307537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:37933 (local address 127.6.22.66:58053)
0131 03:03:03.307748 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.307765 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:03.307888 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:03.308204 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:03.308207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:03.308226 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:03.308457 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:03.308463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:03.309245 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:03.309249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:03.309894 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:03.309908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:03.310032 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:03.310057 (+ 25us) client_negotiation.cc:769] Sending connection context
0131 03:03:03.310115 (+ 58us) client_negotiation.cc:240] Negotiation successful
0131 03:03:03.310214 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260131 03:03:03.310562 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.307545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:40815 (local address 127.6.22.66:44753)
0131 03:03:03.307965 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.307977 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:03.308105 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:03.308392 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:03.308395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:03.308412 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:03.308635 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:03.308642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:03.309448 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:03.309454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:03.310087 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:03.310096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:03.310208 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:03.310225 (+ 17us) client_negotiation.cc:769] Sending connection context
0131 03:03:03.310275 (+ 50us) client_negotiation.cc:240] Negotiation successful
0131 03:03:03.310330 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":38,"thread_start_us":126,"threads_started":1}
I20260131 03:03:03.310889 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.307658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:58053 (local address 127.6.22.67:37933)
0131 03:03:03.307809 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:03.307816 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:03.307836 (+ 20us) server_negotiation.cc:408] Connection header received
0131 03:03:03.308037 (+ 201us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:03.308041 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:03.308090 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:03.308190 (+ 100us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:03.308591 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:03.309113 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:03.310051 (+ 938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:03.310557 (+ 506us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:03.310604 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:03.310663 (+ 59us) server_negotiation.cc:300] Negotiation successful
0131 03:03:03.310735 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260131 03:03:03.310981 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.307750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:44753 (local address 127.6.22.65:40815)
0131 03:03:03.307903 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:03.307907 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:03.308066 (+ 159us) server_negotiation.cc:408] Connection header received
0131 03:03:03.308231 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:03.308234 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:03.308283 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:03.308380 (+ 97us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:03.308815 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:03.309322 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:03.310224 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:03.310699 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:03.310738 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:03.310793 (+ 55us) server_negotiation.cc:300] Negotiation successful
0131 03:03:03.310869 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260131 03:03:03.311437 13928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dadfa1fe33c848288f5ee26f567faf82" candidate_uuid: "256c3d201c82434cb944e16db43d2674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec07f75ac0344b89969f8353b215e19" is_pre_election: true
I20260131 03:03:03.311460 14201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dadfa1fe33c848288f5ee26f567faf82" candidate_uuid: "256c3d201c82434cb944e16db43d2674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f40d5fb02a49a6828f298950421a75" is_pre_election: true
I20260131 03:03:03.311617 13928 raft_consensus.cc:2468] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 256c3d201c82434cb944e16db43d2674 in term 0.
I20260131 03:03:03.311734 14201 raft_consensus.cc:2468] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 256c3d201c82434cb944e16db43d2674 in term 0.
I20260131 03:03:03.312049 13998 leader_election.cc:304] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 256c3d201c82434cb944e16db43d2674, dec07f75ac0344b89969f8353b215e19; no voters:
I20260131 03:03:03.312319 14352 raft_consensus.cc:2804] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:03:03.312402 14352 raft_consensus.cc:493] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:03:03.312458 14352 raft_consensus.cc:3060] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:03.313516 14352 raft_consensus.cc:515] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } } peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } }
I20260131 03:03:03.313927 14352 leader_election.cc:290] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [CANDIDATE]: Term 1 election: Requested vote from peers dec07f75ac0344b89969f8353b215e19 (127.6.22.65:40815), 35f40d5fb02a49a6828f298950421a75 (127.6.22.67:37933)
I20260131 03:03:03.314332 13928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dadfa1fe33c848288f5ee26f567faf82" candidate_uuid: "256c3d201c82434cb944e16db43d2674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec07f75ac0344b89969f8353b215e19"
I20260131 03:03:03.314332 14201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dadfa1fe33c848288f5ee26f567faf82" candidate_uuid: "256c3d201c82434cb944e16db43d2674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f40d5fb02a49a6828f298950421a75"
I20260131 03:03:03.314443 13928 raft_consensus.cc:3060] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:03.314477 14201 raft_consensus.cc:3060] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:03.315367 13928 raft_consensus.cc:2468] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 256c3d201c82434cb944e16db43d2674 in term 1.
I20260131 03:03:03.315531 14201 raft_consensus.cc:2468] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 256c3d201c82434cb944e16db43d2674 in term 1.
I20260131 03:03:03.315711 13998 leader_election.cc:304] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 256c3d201c82434cb944e16db43d2674, dec07f75ac0344b89969f8353b215e19; no voters:
I20260131 03:03:03.315927 14352 raft_consensus.cc:2804] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:03:03.316124 14352 raft_consensus.cc:697] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 1 LEADER]: Becoming Leader. State: Replica: 256c3d201c82434cb944e16db43d2674, State: Running, Role: LEADER
I20260131 03:03:03.316423 14352 consensus_queue.cc:237] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } } peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } }
I20260131 03:03:03.316955 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.316703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.66:55719)
0131 03:03:03.316808 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.316862 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260131 03:03:03.319073 13591 catalog_manager.cc:5654] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 reported cstate change: term changed from 0 to 1, leader changed from <none> to 256c3d201c82434cb944e16db43d2674 (127.6.22.66). New cstate: current_term: 1 leader_uuid: "256c3d201c82434cb944e16db43d2674" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "256c3d201c82434cb944e16db43d2674" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 } health_report { overall_health: UNKNOWN } } }
I20260131 03:03:03.362840 14342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.359657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.6.22.66:44517)
0131 03:03:03.359849 (+ 192us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:03.359853 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:03.359867 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:03:03.359951 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:03.359954 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:03.360009 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:03.360096 (+ 87us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:03:03.360454 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:03.361060 (+ 606us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:03.361812 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:03.361953 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:03.362006 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:03:03.362476 (+ 470us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:03:03.362574 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:03.362678 (+ 104us) server_negotiation.cc:300] Negotiation successful
0131 03:03:03.362729 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260131 03:03:03.365794 13928 raft_consensus.cc:1275] T dadfa1fe33c848288f5ee26f567faf82 P dec07f75ac0344b89969f8353b215e19 [term 1 FOLLOWER]: Refusing update from remote peer 256c3d201c82434cb944e16db43d2674: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:03.366039 14201 raft_consensus.cc:1275] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75 [term 1 FOLLOWER]: Refusing update from remote peer 256c3d201c82434cb944e16db43d2674: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:03.366386 14352 consensus_queue.cc:1048] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec07f75ac0344b89969f8353b215e19" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:03.366672 14355 consensus_queue.cc:1048] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f40d5fb02a49a6828f298950421a75" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:03.369168 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.368852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.67:33319)
0131 03:03:03.369016 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.369085 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260131 03:03:03.370507 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.367992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.65:57045)
0131 03:03:03.370368 (+ 2376us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.370426 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2294}
I20260131 03:03:03.371766 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.371486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.66:36089)
0131 03:03:03.371635 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.371689 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260131 03:03:03.373517 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.373265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.67:58417)
0131 03:03:03.373416 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.373456 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260131 03:03:03.381141 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.380792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.66:60335)
0131 03:03:03.380986 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.381052 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260131 03:03:03.381703 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.381415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:36467 (local address 127.6.22.65:54141)
0131 03:03:03.381577 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.381627 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:36467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260131 03:03:03.385988 13590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:03:03.386219 13590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:03:03.390622 13590 catalog_manager.cc:5941] T 00000000000000000000000000000000 P f4958e667bad4512a95edb0f583dee0a: Sending DeleteTablet for 3 replicas of tablet dadfa1fe33c848288f5ee26f567faf82
I20260131 03:03:03.391356 13908 tablet_service.cc:1552] Processing DeleteTablet for tablet dadfa1fe33c848288f5ee26f567faf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:03 UTC) from {username='oryx', principal='oryx/127.6.22.125@KRBTEST.COM'} at 127.0.0.1:44650
I20260131 03:03:03.391372 14044 tablet_service.cc:1552] Processing DeleteTablet for tablet dadfa1fe33c848288f5ee26f567faf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:03 UTC) from {username='oryx', principal='oryx/127.6.22.125@KRBTEST.COM'} at 127.0.0.1:49136
I20260131 03:03:03.392421 14181 tablet_service.cc:1552] Processing DeleteTablet for tablet dadfa1fe33c848288f5ee26f567faf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:03 UTC) from {username='oryx', principal='oryx/127.6.22.125@KRBTEST.COM'} at 127.0.0.1:32838
I20260131 03:03:03.392498 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 13845
I20260131 03:03:03.395051 14368 tablet_replica.cc:333] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674: stopping tablet replica
I20260131 03:03:03.395190 14368 raft_consensus.cc:2243] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:03:03.395530 14368 raft_consensus.cc:2272] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:03.395888 14370 tablet_replica.cc:333] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75: stopping tablet replica
I20260131 03:03:03.396243 14370 raft_consensus.cc:2243] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:03:03.396518 14370 raft_consensus.cc:2272] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:03.397532 14370 ts_tablet_manager.cc:1916] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:03.399860 14370 ts_tablet_manager.cc:1929] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:03:03.399967 14370 log.cc:1199] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/dadfa1fe33c848288f5ee26f567faf82
I20260131 03:03:03.400269 14370 ts_tablet_manager.cc:1950] T dadfa1fe33c848288f5ee26f567faf82 P 35f40d5fb02a49a6828f298950421a75: Deleting consensus metadata
I20260131 03:03:03.401257 14368 ts_tablet_manager.cc:1916] T dadfa1fe33c848288f5ee26f567faf82 P 256c3d201c82434cb944e16db43d2674: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:03.401324 13577 catalog_manager.cc:4985] TS 35f40d5fb02a49a6828f298950421a75 (127.6.22.67:37933): tablet dadfa1fe33c848288f5ee26f567faf82 (table test-table [id=29fb508aae83456fa0a03e5c2ec10392]) successfully deleted
W20260131 03:03:03.403050 14341 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:03.402201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:40815 (local address 127.0.0.1:44662)
0131 03:03:03.402379 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.402399 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:03.402529 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:03.402926 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.65:40815: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":68}
W20260131 03:03:03.403262 13576 catalog_manager.cc:4712] TS dec07f75ac0344b89969f8353b215e19 (127.6.22.65:40815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dadfa1fe33c848288f5ee26f567faf82: Network error: Client connection negotiation failed: client connection to 127.6.22.65:40815: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260131 03:03:03.403347 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 13981
I20260131 03:03:03.410487 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 14117
I20260131 03:03:03.411633 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:03.410588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:44517 (local address 127.0.0.1:49152)
0131 03:03:03.411496 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:03.411547 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.66:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":816}
I20260131 03:03:03.417943 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 13453
I20260131 03:03:03.424633 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 13555
2026-01-31T03:03:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10257 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:03:03 dist-test-slave-j368 krb5kdc[14373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:03:03 dist-test-slave-j368 krb5kdc[14373](info): set up 2 sockets
Jan 31 03:03:03 dist-test-slave-j368 krb5kdc[14373](info): commencing operation
krb5kdc: starting...
W20260131 03:03:05.479583 6233 mini_kdc.cc:121] Time spent starting KDC: real 2.029s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:03:05 dist-test-slave-j368 krb5kdc[14373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:03:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:03:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
I20260131 03:03:05.629266 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:46517
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:38645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:46517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:05.741003 14389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:05.741258 14389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:05.741372 14389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:05.744910 14389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:03:05.744989 14389 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260131 03:03:05.745008 14389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:05.745083 14389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:05.745121 14389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:03:05.745162 14389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:03:05.749768 14389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:38645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:46517
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:46517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:05.751008 14389 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:05.751904 14389 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:05.758147 14394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:05.758150 14397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:05.758147 14395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:05.758430 14389 server_base.cc:1060] running on GCE node
I20260131 03:03:05.759011 14389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:05.760022 14389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:05.761322 14389 hybrid_clock.cc:648] HybridClock initialized: now 1769828585761274 us; error 88 us; skew 500 ppm
Jan 31 03:03:05 dist-test-slave-j368 krb5kdc[14373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828585, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:05.764534 14389 init.cc:377] Logged in from keytab as oryx/127.6.22.126@KRBTEST.COM (short username oryx)
I20260131 03:03:05.766008 14389 webserver.cc:492] Webserver started at http://127.6.22.126:40231/ using document root <none> and password file <none>
I20260131 03:03:05.766659 14389 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:05.766749 14389 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:05.766990 14389 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:05.769011 14389 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "650af9a4f6f14165bb9971ba0813c28d"
format_stamp: "Formatted at 2026-01-31 03:03:05 on dist-test-slave-j368"
server_key: "a36fb3c42ae30c9109fa18d64ab34a21"
server_key_iv: "482127821aae91df0d2afe0957916dfe"
server_key_version: "encryptionkey@0"
I20260131 03:03:05.769579 14389 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "650af9a4f6f14165bb9971ba0813c28d"
format_stamp: "Formatted at 2026-01-31 03:03:05 on dist-test-slave-j368"
server_key: "a36fb3c42ae30c9109fa18d64ab34a21"
server_key_iv: "482127821aae91df0d2afe0957916dfe"
server_key_version: "encryptionkey@0"
I20260131 03:03:05.773526 14389 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260131 03:03:05.776408 14404 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:05.777832 14389 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260131 03:03:05.778028 14389 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "650af9a4f6f14165bb9971ba0813c28d"
format_stamp: "Formatted at 2026-01-31 03:03:05 on dist-test-slave-j368"
server_key: "a36fb3c42ae30c9109fa18d64ab34a21"
server_key_iv: "482127821aae91df0d2afe0957916dfe"
server_key_version: "encryptionkey@0"
I20260131 03:03:05.778158 14389 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:05.795158 14389 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:05.798249 14389 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:05.798673 14389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:05.807657 14389 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:46517
I20260131 03:03:05.807932 14466 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:46517 every 8 connection(s)
I20260131 03:03:05.808852 14389 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:03:05.811728 14467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:05.816162 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 14389
I20260131 03:03:05.816329 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:03:05.816570 6233 external_mini_cluster.cc:1468] Setting key 894599ee00c926bb23d032fc6099600b
I20260131 03:03:05.817221 14467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d: Bootstrap starting.
I20260131 03:03:05.819873 14467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:05.820763 14467 log.cc:826] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d: Log is configured to *not* fsync() on all Append() calls
I20260131 03:03:05.822841 14467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d: No bootstrap required, opened a new log
Jan 31 03:03:05 dist-test-slave-j368 krb5kdc[14373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:03:05.826334 14467 raft_consensus.cc:359] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650af9a4f6f14165bb9971ba0813c28d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 46517 } }
I20260131 03:03:05.826705 14467 raft_consensus.cc:385] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:05.826781 14467 raft_consensus.cc:740] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 650af9a4f6f14165bb9971ba0813c28d, State: Initialized, Role: FOLLOWER
I20260131 03:03:05.827286 14467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650af9a4f6f14165bb9971ba0813c28d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 46517 } }
I20260131 03:03:05.827452 14467 raft_consensus.cc:399] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:03:05.827526 14467 raft_consensus.cc:493] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:03:05.827625 14467 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:05.828860 14467 raft_consensus.cc:515] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650af9a4f6f14165bb9971ba0813c28d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 46517 } }
I20260131 03:03:05.829226 14467 leader_election.cc:304] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 650af9a4f6f14165bb9971ba0813c28d; no voters:
I20260131 03:03:05.829586 14467 leader_election.cc:290] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:03:05.829869 14472 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:03:05.830109 14472 raft_consensus.cc:697] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [term 1 LEADER]: Becoming Leader. State: Replica: 650af9a4f6f14165bb9971ba0813c28d, State: Running, Role: LEADER
I20260131 03:03:05.830448 14472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650af9a4f6f14165bb9971ba0813c28d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 46517 } }
I20260131 03:03:05.830564 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:05.817995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.6.22.126:46517)
0131 03:03:05.818460 (+ 465us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:05.818475 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:05.818521 (+ 46us) server_negotiation.cc:408] Connection header received
0131 03:03:05.819220 (+ 699us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:05.819243 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:05.819605 (+ 362us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:05.819991 (+ 386us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:05.820897 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:05.821904 (+ 1007us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:05.822625 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:05.822902 (+ 277us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:05.825365 (+ 2463us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:05.825392 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:05.825406 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:05.825453 (+ 47us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:05.827631 (+ 2178us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:05.828165 (+ 534us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:05.828170 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:05.828174 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:05.828253 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:05.828507 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:05.828510 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:05.828511 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:05.828914 (+ 403us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:05.829183 (+ 269us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:05.829468 (+ 285us) server_negotiation.cc:300] Negotiation successful
0131 03:03:05.829760 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":123,"threads_started":1}
I20260131 03:03:05.830845 14467 sys_catalog.cc:565] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:03:05.832031 14474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 650af9a4f6f14165bb9971ba0813c28d. Latest consensus state: current_term: 1 leader_uuid: "650af9a4f6f14165bb9971ba0813c28d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650af9a4f6f14165bb9971ba0813c28d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 46517 } } }
I20260131 03:03:05.832149 14474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:05.832429 14473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "650af9a4f6f14165bb9971ba0813c28d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650af9a4f6f14165bb9971ba0813c28d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 46517 } } }
I20260131 03:03:05.832520 14473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:05.833047 14481 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:03:05.836079 14481 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:03:05.842274 14481 catalog_manager.cc:1357] Generated new cluster ID: 72e42a4a87e94e599b75a43f37454fb3
I20260131 03:03:05.842366 14481 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:03:05.874611 14481 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:03:05.875916 14481 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:03:05.886262 14481 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d: Generated new TSK 0
I20260131 03:03:05.887085 14481 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
I20260131 03:03:05.947582 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:46517
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:38645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:06.076819 14495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:06.077068 14495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:06.077135 14495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:06.080528 14495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:06.080619 14495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:06.080701 14495 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:03:06.080746 14495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:03:06.085439 14495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:38645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.22.126:46517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:06.086589 14495 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:06.087440 14495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:06.094460 14503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:06.094424 14501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:06.094424 14500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:06.094662 14495 server_base.cc:1060] running on GCE node
I20260131 03:03:06.095132 14495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:06.095669 14495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:06.096872 14495 hybrid_clock.cc:648] HybridClock initialized: now 1769828586096844 us; error 46 us; skew 500 ppm
Jan 31 03:03:06 dist-test-slave-j368 krb5kdc[14373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828586, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:06.099709 14495 init.cc:377] Logged in from keytab as oryx/127.6.22.65@KRBTEST.COM (short username oryx)
I20260131 03:03:06.100932 14495 webserver.cc:492] Webserver started at http://127.6.22.65:34097/ using document root <none> and password file <none>
I20260131 03:03:06.101524 14495 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:06.101583 14495 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:06.101796 14495 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:06.103538 14495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "f8e846a385ec4544ad7ecde4a1336cf8"
format_stamp: "Formatted at 2026-01-31 03:03:06 on dist-test-slave-j368"
server_key: "38684f41876ec0670124b4990133697d"
server_key_iv: "cc7a1fec8a38b131c82af2b1e7505c04"
server_key_version: "encryptionkey@0"
I20260131 03:03:06.104115 14495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "f8e846a385ec4544ad7ecde4a1336cf8"
format_stamp: "Formatted at 2026-01-31 03:03:06 on dist-test-slave-j368"
server_key: "38684f41876ec0670124b4990133697d"
server_key_iv: "cc7a1fec8a38b131c82af2b1e7505c04"
server_key_version: "encryptionkey@0"
I20260131 03:03:06.107736 14495 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260131 03:03:06.110095 14510 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:06.111199 14495 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260131 03:03:06.111349 14495 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "f8e846a385ec4544ad7ecde4a1336cf8"
format_stamp: "Formatted at 2026-01-31 03:03:06 on dist-test-slave-j368"
server_key: "38684f41876ec0670124b4990133697d"
server_key_iv: "cc7a1fec8a38b131c82af2b1e7505c04"
server_key_version: "encryptionkey@0"
I20260131 03:03:06.111472 14495 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:06.129021 14495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:06.132277 14495 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:06.132537 14495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:06.134168 14495 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:06.134258 14495 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:06.134317 14495 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:06.134346 14495 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:03:06 dist-test-slave-j368 krb5kdc[14373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828586, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:03:06.146857 14495 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:44813
I20260131 03:03:06.146950 14625 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:44813 every 8 connection(s)
I20260131 03:03:06.147997 14495 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:03:06.149916 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.134920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:49021 (local address 127.6.22.126:46517)
0131 03:03:06.135088 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.135095 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.135839 (+ 744us) server_negotiation.cc:408] Connection header received
0131 03:03:06.136875 (+ 1036us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.136880 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.136950 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.137064 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:06.138960 (+ 1896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.139985 (+ 1025us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.140956 (+ 971us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.141170 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.145247 (+ 4077us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:06.145290 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:06.145293 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:06.145334 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:06.147245 (+ 1911us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.148035 (+ 790us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.148040 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.148042 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.148191 (+ 149us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.148586 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.148590 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.148592 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.148897 (+ 305us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:06.149027 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.149556 (+ 529us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.149744 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260131 03:03:06.150492 14519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.135232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:46517 (local address 127.6.22.65:49021)
0131 03:03:06.135673 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:06.135715 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:06.136592 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:06.137337 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:06.137356 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:06.138011 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:06.138640 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.138679 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.140234 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.140239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:06.140806 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.140819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.141163 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.142288 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:06.142354 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:06.144962 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:06.147421 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:06.147428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:06.147438 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:06.147825 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:06.148321 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:06.148327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:06.148330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:06.148461 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:06.149024 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:06.149031 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:06.149293 (+ 262us) client_negotiation.cc:769] Sending connection context
0131 03:03:06.149505 (+ 212us) client_negotiation.cc:240] Negotiation successful
0131 03:03:06.149796 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":138,"threads_started":1}
I20260131 03:03:06.151859 14626 heartbeater.cc:344] Connected to a master server at 127.6.22.126:46517
I20260131 03:03:06.152089 14626 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:06.152626 14626 heartbeater.cc:507] Master 127.6.22.126:46517 requested a full tablet report, sending...
I20260131 03:03:06.154393 14421 ts_manager.cc:194] Registered new tserver with Master: f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65:44813)
I20260131 03:03:06.154883 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 14495
I20260131 03:03:06.155027 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:03:06.155278 6233 external_mini_cluster.cc:1468] Setting key 1242656bad44ea4d2b0e9eb32b194357
I20260131 03:03:06.156047 14421 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:49021
I20260131 03:03:06.162417 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.154247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:46517 (local address 127.6.22.65:57721)
0131 03:03:06.154681 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:06.154700 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:06.154812 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:06.155163 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:06.155166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:06.155383 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:06.155688 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.155697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.156604 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.156609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:06.157230 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.157240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.157721 (+ 481us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.158335 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:06.158356 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:06.158813 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:06.160983 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:06.160987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:06.160989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:06.161191 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:06.161501 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:06.161504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:06.161505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:06.161554 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:06.161984 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:06.161986 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:06.162068 (+ 82us) client_negotiation.cc:769] Sending connection context
0131 03:03:06.162149 (+ 81us) client_negotiation.cc:240] Negotiation successful
0131 03:03:06.162264 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":111,"threads_started":1}
I20260131 03:03:06.162643 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.154320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:57721 (local address 127.6.22.126:46517)
0131 03:03:06.154479 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.154485 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.154784 (+ 299us) server_negotiation.cc:408] Connection header received
0131 03:03:06.154912 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.154915 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.154962 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.155047 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:06.155866 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.156482 (+ 616us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.157369 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.157587 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.158983 (+ 1396us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:06.159006 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:06.159013 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:06.159047 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:06.160855 (+ 1808us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.161321 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.161329 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.161332 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.161398 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.161660 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.161664 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.161666 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.161884 (+ 218us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:06.161997 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.162249 (+ 252us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.162395 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for oryx/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
I20260131 03:03:06.220357 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:46517
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:38645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:06.333231 14637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:06.333560 14637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:06.333642 14637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:06.337066 14637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:06.337137 14637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:06.337193 14637 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:03:06.337241 14637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:03:06.341993 14637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:38645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.22.126:46517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:06.343158 14637 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:06.343990 14637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:06.352788 14642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:06.352788 14645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:06.352834 14643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:06.353176 14637 server_base.cc:1060] running on GCE node
I20260131 03:03:06.353683 14637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:06.354308 14637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:06.355499 14637 hybrid_clock.cc:648] HybridClock initialized: now 1769828586355479 us; error 48 us; skew 500 ppm
Jan 31 03:03:06 dist-test-slave-j368 krb5kdc[14373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828586, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:06.358385 14637 init.cc:377] Logged in from keytab as oryx/127.6.22.66@KRBTEST.COM (short username oryx)
I20260131 03:03:06.359804 14637 webserver.cc:492] Webserver started at http://127.6.22.66:39245/ using document root <none> and password file <none>
I20260131 03:03:06.360929 14637 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:06.361044 14637 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:06.361259 14637 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:06.363121 14637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "e6de748b72ca40209a51efb372bf1cfb"
format_stamp: "Formatted at 2026-01-31 03:03:06 on dist-test-slave-j368"
server_key: "f871a161bc59e63b268e044e9b7ada50"
server_key_iv: "3993b54ff10473bf77884ea17660fa3a"
server_key_version: "encryptionkey@0"
I20260131 03:03:06.363662 14637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "e6de748b72ca40209a51efb372bf1cfb"
format_stamp: "Formatted at 2026-01-31 03:03:06 on dist-test-slave-j368"
server_key: "f871a161bc59e63b268e044e9b7ada50"
server_key_iv: "3993b54ff10473bf77884ea17660fa3a"
server_key_version: "encryptionkey@0"
I20260131 03:03:06.367566 14637 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260131 03:03:06.370069 14652 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:06.371145 14637 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:03:06.371299 14637 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "e6de748b72ca40209a51efb372bf1cfb"
format_stamp: "Formatted at 2026-01-31 03:03:06 on dist-test-slave-j368"
server_key: "f871a161bc59e63b268e044e9b7ada50"
server_key_iv: "3993b54ff10473bf77884ea17660fa3a"
server_key_version: "encryptionkey@0"
I20260131 03:03:06.371423 14637 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:06.390635 14637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:06.393918 14637 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:06.394150 14637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:06.395641 14637 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:06.395726 14637 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:06.395792 14637 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:06.395844 14637 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:03:06 dist-test-slave-j368 krb5kdc[14373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828586, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:03:06.409147 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.396542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:46491 (local address 127.6.22.126:46517)
0131 03:03:06.396738 (+ 196us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.396750 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.397966 (+ 1216us) server_negotiation.cc:408] Connection header received
0131 03:03:06.398682 (+ 716us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.398687 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.398786 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.398981 (+ 195us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:06.400745 (+ 1764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.401341 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.402059 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.402238 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.405091 (+ 2853us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:06.405114 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:06.405116 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:06.405147 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:06.406661 (+ 1514us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.407202 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.407206 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.407210 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.407259 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.407625 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.407631 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.407634 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.407913 (+ 279us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:06.408019 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.408814 (+ 795us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.408965 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260131 03:03:06.409885 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.396863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:46517 (local address 127.6.22.66:46491)
0131 03:03:06.397331 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:06.397374 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:06.398381 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:06.399203 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:06.399214 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:06.399677 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:06.400444 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.400459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.401487 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.401496 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:06.401925 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.401932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.402210 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.403061 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:06.403106 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:06.404899 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:06.406814 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:06.406819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:06.406833 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:06.407087 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:06.407356 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:06.407360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:06.407363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:06.407483 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:06.408088 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:06.408097 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:06.408484 (+ 387us) client_negotiation.cc:769] Sending connection context
0131 03:03:06.408779 (+ 295us) client_negotiation.cc:240] Negotiation successful
0131 03:03:06.409070 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":151,"threads_started":1}
I20260131 03:03:06.409992 14637 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:41925
I20260131 03:03:06.410034 14767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:41925 every 8 connection(s)
I20260131 03:03:06.410970 14637 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:03:06.412467 14768 heartbeater.cc:344] Connected to a master server at 127.6.22.126:46517
I20260131 03:03:06.413012 14768 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:06.413596 14768 heartbeater.cc:507] Master 127.6.22.126:46517 requested a full tablet report, sending...
I20260131 03:03:06.414613 14420 ts_manager.cc:194] Registered new tserver with Master: e6de748b72ca40209a51efb372bf1cfb (127.6.22.66:41925)
I20260131 03:03:06.415334 14420 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:46491
I20260131 03:03:06.418460 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 14637
I20260131 03:03:06.418741 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:03:06.419188 6233 external_mini_cluster.cc:1468] Setting key d25b8b4b9673cc110ca42e64b150f07a
I20260131 03:03:06.422103 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.413641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:46517 (local address 127.6.22.66:48151)
0131 03:03:06.413950 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:06.413966 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:06.414124 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:06.414419 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:06.414421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:06.414589 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:06.414864 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.414872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.415823 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.415826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:06.416313 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.416322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.416455 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.417091 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:06.417105 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:06.417403 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:06.420127 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:06.420132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:06.420134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:06.420458 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:06.420869 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:06.420873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:06.420875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:06.420946 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:06.421546 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:06.421549 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:06.421658 (+ 109us) client_negotiation.cc:769] Sending connection context
0131 03:03:06.421762 (+ 104us) client_negotiation.cc:240] Negotiation successful
0131 03:03:06.421922 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":94,"threads_started":1}
I20260131 03:03:06.422266 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.413749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:48151 (local address 127.6.22.126:46517)
0131 03:03:06.413903 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.413907 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.414043 (+ 136us) server_negotiation.cc:408] Connection header received
0131 03:03:06.414220 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.414223 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.414302 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.414389 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:06.414995 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.415720 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.416448 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.416693 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.417572 (+ 879us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:06.417589 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:06.417597 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:06.417619 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:06.419900 (+ 2281us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.420631 (+ 731us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.420634 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.420636 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.420750 (+ 114us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.421065 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.421068 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.421070 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.421373 (+ 303us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:06.421483 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.421869 (+ 386us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.422016 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for oryx/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab.
I20260131 03:03:06.476279 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:46517
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:38645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:06.587220 14779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:06.587476 14779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:06.587541 14779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:06.591034 14779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:06.591120 14779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:06.591177 14779 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:03:06.591220 14779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:03:06.595831 14779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:38645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.22.126:46517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:06.596982 14779 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:06.597851 14779 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:06.605182 14784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:06.605187 14787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:06.605182 14785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:06.605580 14779 server_base.cc:1060] running on GCE node
I20260131 03:03:06.605998 14779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:06.606630 14779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:06.607834 14779 hybrid_clock.cc:648] HybridClock initialized: now 1769828586607818 us; error 39 us; skew 500 ppm
Jan 31 03:03:06 dist-test-slave-j368 krb5kdc[14373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828586, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:06.610893 14779 init.cc:377] Logged in from keytab as oryx/127.6.22.67@KRBTEST.COM (short username oryx)
I20260131 03:03:06.612123 14779 webserver.cc:492] Webserver started at http://127.6.22.67:38617/ using document root <none> and password file <none>
I20260131 03:03:06.612823 14779 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:06.613301 14779 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:06.613543 14779 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:06.615370 14779 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "6110ec810fd74ff2adf98a640f84b2e6"
format_stamp: "Formatted at 2026-01-31 03:03:06 on dist-test-slave-j368"
server_key: "c0b8528e1a1a6410fe7ea93f6362b0c8"
server_key_iv: "b3cdc4f3faebf3ed09aa8812ad3c55fa"
server_key_version: "encryptionkey@0"
I20260131 03:03:06.615932 14779 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "6110ec810fd74ff2adf98a640f84b2e6"
format_stamp: "Formatted at 2026-01-31 03:03:06 on dist-test-slave-j368"
server_key: "c0b8528e1a1a6410fe7ea93f6362b0c8"
server_key_iv: "b3cdc4f3faebf3ed09aa8812ad3c55fa"
server_key_version: "encryptionkey@0"
I20260131 03:03:06.619992 14779 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260131 03:03:06.622840 14794 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:06.624253 14779 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260131 03:03:06.624429 14779 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "6110ec810fd74ff2adf98a640f84b2e6"
format_stamp: "Formatted at 2026-01-31 03:03:06 on dist-test-slave-j368"
server_key: "c0b8528e1a1a6410fe7ea93f6362b0c8"
server_key_iv: "b3cdc4f3faebf3ed09aa8812ad3c55fa"
server_key_version: "encryptionkey@0"
I20260131 03:03:06.624553 14779 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:06.638638 14779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:06.643173 14779 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:06.643447 14779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:06.645180 14779 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:06.645275 14779 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:06.645344 14779 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:06.645382 14779 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:03:06 dist-test-slave-j368 krb5kdc[14373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828586, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:03:06.657483 14779 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:33951
I20260131 03:03:06.657500 14909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:33951 every 8 connection(s)
I20260131 03:03:06.658483 14779 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:03:06.658526 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.645884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:32989 (local address 127.6.22.126:46517)
0131 03:03:06.646068 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.646073 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.646934 (+ 861us) server_negotiation.cc:408] Connection header received
0131 03:03:06.648052 (+ 1118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.648057 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.648128 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.648209 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:06.650324 (+ 2115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.651079 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.651818 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.652042 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.654739 (+ 2697us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:06.654763 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:06.654766 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:06.654799 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:06.656364 (+ 1565us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.656953 (+ 589us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.656957 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.656959 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.657016 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.657336 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.657339 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.657341 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.657510 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:06.657640 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.658225 (+ 585us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.658349 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260131 03:03:06.659222 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.646209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:46517 (local address 127.6.22.67:32989)
0131 03:03:06.646755 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:06.646809 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:06.647799 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:06.648417 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:06.648432 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:06.649134 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:06.649961 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.649981 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.651214 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.651218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:06.651694 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.651707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.651950 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.652754 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:06.652790 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:06.654571 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:06.656490 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:06.656497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:06.656512 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:06.656846 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:06.657117 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:06.657120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:06.657122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:06.657243 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:06.657640 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:06.657648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:06.657965 (+ 317us) client_negotiation.cc:769] Sending connection context
0131 03:03:06.658174 (+ 209us) client_negotiation.cc:240] Negotiation successful
0131 03:03:06.658440 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":107,"threads_started":1}
I20260131 03:03:06.660526 14910 heartbeater.cc:344] Connected to a master server at 127.6.22.126:46517
I20260131 03:03:06.660737 14910 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:06.661239 14910 heartbeater.cc:507] Master 127.6.22.126:46517 requested a full tablet report, sending...
I20260131 03:03:06.662278 14421 ts_manager.cc:194] Registered new tserver with Master: 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951)
I20260131 03:03:06.662772 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 14779
I20260131 03:03:06.662874 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:03:06.663086 14421 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:32989
I20260131 03:03:06.663151 6233 external_mini_cluster.cc:1468] Setting key ea9278a430304e3ad454831549489ae2
I20260131 03:03:06.665148 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:03:06.669893 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.662625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:34121 (local address 127.6.22.126:46517)
0131 03:03:06.662751 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.662756 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.662983 (+ 227us) server_negotiation.cc:408] Connection header received
0131 03:03:06.663163 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.663166 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.663214 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.663301 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:06.664030 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.664613 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.665489 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.665698 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.666555 (+ 857us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:06.666576 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:06.666583 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:06.666616 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:06.668167 (+ 1551us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.668733 (+ 566us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.668737 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.668739 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.668780 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.669080 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.669083 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.669084 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.669232 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:06.669311 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.669617 (+ 306us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.669720 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260131 03:03:06.670006 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.662585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:46517 (local address 127.6.22.67:34121)
0131 03:03:06.662902 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:06.662918 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:06.663054 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:06.663436 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:06.663439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:06.663644 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:06.663922 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.663931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.664809 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.664813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:06.665375 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.665384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.665506 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.666129 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:06.666142 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:06.666421 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:06.668306 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:06.668312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:06.668317 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:06.668558 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:06.668885 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:06.668891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:06.668893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:06.668965 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:06.669335 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:06.669338 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:06.669449 (+ 111us) client_negotiation.cc:769] Sending connection context
0131 03:03:06.669568 (+ 119us) client_negotiation.cc:240] Negotiation successful
0131 03:03:06.669732 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":84,"threads_started":1}
Jan 31 03:03:06 dist-test-slave-j368 krb5kdc[14373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 31 03:03:06 dist-test-slave-j368 krb5kdc[14373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:06 dist-test-slave-j368 krb5kdc[14373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
W20260131 03:03:06.690199 14470 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:06.683547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.6.22.126:46517)
0131 03:03:06.683736 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.683741 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.683760 (+ 19us) server_negotiation.cc:408] Connection header received
0131 03:03:06.683885 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.683888 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.683941 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.684038 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:06.685140 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.685657 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.686451 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.686577 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.690000 (+ 3423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51610: BlockingRecv error: recv got EOF from 127.0.0.1:51610 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jan 31 03:03:06 dist-test-slave-j368 krb5kdc[14373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:03:06.699560 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.691843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51624 (local address 127.6.22.126:46517)
0131 03:03:06.691994 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.691999 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.692073 (+ 74us) server_negotiation.cc:408] Connection header received
0131 03:03:06.692244 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.692252 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.692298 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.692391 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:06.693271 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.693786 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.694423 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.694632 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.696684 (+ 2052us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:06.696711 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:06.696714 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:06.696750 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:06.697954 (+ 1204us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.698368 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.698373 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.698375 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.698425 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:06.698710 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:06.698716 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:06.698719 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:06.698889 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:06.699047 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.699268 (+ 221us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.699381 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260131 03:03:06.702359 14421 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:03:06.704968 14421 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:03:06.718241 14932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.713549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:44813 (local address 127.0.0.1:57384)
0131 03:03:06.713927 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:06.713962 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:06.714076 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:06.714739 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:06.714744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:06.714781 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:06.715130 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.715139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.716773 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.716778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:06.717769 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.717778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.717859 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.717919 (+ 60us) client_negotiation.cc:769] Sending connection context
0131 03:03:06.717999 (+ 80us) client_negotiation.cc:240] Negotiation successful
0131 03:03:06.718088 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
I20260131 03:03:06.718654 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.714532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:41925 (local address 127.0.0.1:49936)
0131 03:03:06.714970 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:06.714984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:06.715066 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:06.715742 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:06.715745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:06.715762 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:06.716023 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.716031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.717426 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.717430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:06.718302 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.718315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.718392 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.718413 (+ 21us) client_negotiation.cc:769] Sending connection context
0131 03:03:06.718458 (+ 45us) client_negotiation.cc:240] Negotiation successful
0131 03:03:06.718511 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":84,"threads_started":1}
I20260131 03:03:06.719358 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.713692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57384 (local address 127.6.22.65:44813)
0131 03:03:06.714254 (+ 562us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.714261 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.714286 (+ 25us) server_negotiation.cc:408] Connection header received
0131 03:03:06.714373 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.714379 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.714559 (+ 180us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.715272 (+ 713us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:06.715405 (+ 133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.716594 (+ 1189us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.718364 (+ 1770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.718938 (+ 574us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.719062 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.719131 (+ 69us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.719219 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":93,"threads_started":1}
I20260131 03:03:06.719358 14935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.714407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.6.22.67:33951)
0131 03:03:06.714788 (+ 381us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.714792 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.715032 (+ 240us) server_negotiation.cc:408] Connection header received
0131 03:03:06.715318 (+ 286us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.715322 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.715443 (+ 121us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.715576 (+ 133us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:06.716002 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.716962 (+ 960us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.717984 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.718938 (+ 954us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.719062 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.719131 (+ 69us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.719219 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":101,"threads_started":1}
I20260131 03:03:06.719511 14937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.714937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.6.22.66:41925)
0131 03:03:06.715332 (+ 395us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.715338 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.715358 (+ 20us) server_negotiation.cc:408] Connection header received
0131 03:03:06.715438 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.715446 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.715593 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.715750 (+ 157us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:06.716144 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.717296 (+ 1152us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.718668 (+ 1372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.719140 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.719223 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.719275 (+ 52us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.719337 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260131 03:03:06.718243 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.714242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:33951 (local address 127.0.0.1:38724)
0131 03:03:06.714945 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:06.714960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:06.715127 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:06.715575 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:06.715578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:06.715595 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:06.715872 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.715879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.717099 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.717102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:06.717832 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.717841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.717976 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.717993 (+ 17us) client_negotiation.cc:769] Sending connection context
0131 03:03:06.718046 (+ 53us) client_negotiation.cc:240] Negotiation successful
0131 03:03:06.718104 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":96,"threads_started":1}
I20260131 03:03:06.722218 14702 tablet_service.cc:1505] Processing CreateTablet for tablet 0237d95169314c52a6d915a736de4404 (DEFAULT_TABLE table=test-table [id=2eeaf8bd9a6d427c8d7ed500787f42eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:06.722258 14558 tablet_service.cc:1505] Processing CreateTablet for tablet 0237d95169314c52a6d915a736de4404 (DEFAULT_TABLE table=test-table [id=2eeaf8bd9a6d427c8d7ed500787f42eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:06.722280 14844 tablet_service.cc:1505] Processing CreateTablet for tablet 0237d95169314c52a6d915a736de4404 (DEFAULT_TABLE table=test-table [id=2eeaf8bd9a6d427c8d7ed500787f42eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:06.723198 14702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0237d95169314c52a6d915a736de4404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:06.723203 14558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0237d95169314c52a6d915a736de4404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:06.723203 14844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0237d95169314c52a6d915a736de4404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:06.729957 14938 tablet_bootstrap.cc:492] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb: Bootstrap starting.
I20260131 03:03:06.730031 14940 tablet_bootstrap.cc:492] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6: Bootstrap starting.
I20260131 03:03:06.730032 14939 tablet_bootstrap.cc:492] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8: Bootstrap starting.
I20260131 03:03:06.732103 14938 tablet_bootstrap.cc:654] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:06.732160 14940 tablet_bootstrap.cc:654] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:06.732160 14939 tablet_bootstrap.cc:654] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:06.733134 14939 log.cc:826] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8: Log is configured to *not* fsync() on all Append() calls
I20260131 03:03:06.733372 14938 log.cc:826] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb: Log is configured to *not* fsync() on all Append() calls
I20260131 03:03:06.733387 14940 log.cc:826] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6: Log is configured to *not* fsync() on all Append() calls
I20260131 03:03:06.735246 14939 tablet_bootstrap.cc:492] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8: No bootstrap required, opened a new log
I20260131 03:03:06.735247 14940 tablet_bootstrap.cc:492] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6: No bootstrap required, opened a new log
I20260131 03:03:06.735445 14939 ts_tablet_manager.cc:1403] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260131 03:03:06.735445 14940 ts_tablet_manager.cc:1403] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260131 03:03:06.736083 14938 tablet_bootstrap.cc:492] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb: No bootstrap required, opened a new log
I20260131 03:03:06.736263 14938 ts_tablet_manager.cc:1403] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260131 03:03:06.738150 14940 raft_consensus.cc:359] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } }
I20260131 03:03:06.738376 14940 raft_consensus.cc:385] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:06.738430 14940 raft_consensus.cc:740] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6110ec810fd74ff2adf98a640f84b2e6, State: Initialized, Role: FOLLOWER
I20260131 03:03:06.738631 14939 raft_consensus.cc:359] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } }
I20260131 03:03:06.738827 14939 raft_consensus.cc:385] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:06.738881 14939 raft_consensus.cc:740] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e846a385ec4544ad7ecde4a1336cf8, State: Initialized, Role: FOLLOWER
I20260131 03:03:06.738903 14940 consensus_queue.cc:260] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } }
I20260131 03:03:06.739296 14939 consensus_queue.cc:260] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } }
I20260131 03:03:06.739264 14938 raft_consensus.cc:359] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } }
I20260131 03:03:06.739439 14938 raft_consensus.cc:385] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:06.739495 14938 raft_consensus.cc:740] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6de748b72ca40209a51efb372bf1cfb, State: Initialized, Role: FOLLOWER
I20260131 03:03:06.739795 14910 heartbeater.cc:499] Master 127.6.22.126:46517 was elected leader, sending a full tablet report...
I20260131 03:03:06.739912 14940 ts_tablet_manager.cc:1434] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260131 03:03:06.739935 14938 consensus_queue.cc:260] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } }
I20260131 03:03:06.740067 14939 ts_tablet_manager.cc:1434] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260131 03:03:06.740373 14626 heartbeater.cc:499] Master 127.6.22.126:46517 was elected leader, sending a full tablet report...
I20260131 03:03:06.740602 14768 heartbeater.cc:499] Master 127.6.22.126:46517 was elected leader, sending a full tablet report...
I20260131 03:03:06.740691 14938 ts_tablet_manager.cc:1434] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb: Time spent starting tablet: real 0.004s user 0.004s sys 0.002s
I20260131 03:03:06.869439 14945 raft_consensus.cc:493] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:03:06.869678 14945 raft_consensus.cc:515] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } }
I20260131 03:03:06.871101 14945 leader_election.cc:290] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951), e6de748b72ca40209a51efb372bf1cfb (127.6.22.66:41925)
I20260131 03:03:06.876238 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.871581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:33951 (local address 127.6.22.65:57185)
0131 03:03:06.872803 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:06.872827 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:06.873068 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:06.873501 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:06.873504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:06.873533 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:06.873881 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.873890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.874838 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.874845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:06.875568 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.875578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.875761 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.875807 (+ 46us) client_negotiation.cc:769] Sending connection context
0131 03:03:06.875870 (+ 63us) client_negotiation.cc:240] Negotiation successful
0131 03:03:06.875952 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"thread_start_us":854,"threads_started":1}
I20260131 03:03:06.876262 14948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.872861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:41925 (local address 127.6.22.65:42839)
0131 03:03:06.873264 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:06.873278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:06.873399 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:06.873725 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:06.873728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:06.873747 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:06.874093 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.874102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.874991 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.874995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:06.875629 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:06.875694 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.875856 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.875875 (+ 19us) client_negotiation.cc:769] Sending connection context
0131 03:03:06.875939 (+ 64us) client_negotiation.cc:240] Negotiation successful
0131 03:03:06.876002 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":104,"threads_started":1}
I20260131 03:03:06.876645 14935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.871769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:57185 (local address 127.6.22.67:33951)
0131 03:03:06.872096 (+ 327us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.872103 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.872947 (+ 844us) server_negotiation.cc:408] Connection header received
0131 03:03:06.873243 (+ 296us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.873248 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.873335 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.873427 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:06.874018 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.874678 (+ 660us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.875785 (+ 1107us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.876382 (+ 597us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.876428 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.876496 (+ 68us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.876554 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260131 03:03:06.876684 14937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.872990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:42839 (local address 127.6.22.66:41925)
0131 03:03:06.873149 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.873155 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.873358 (+ 203us) server_negotiation.cc:408] Connection header received
0131 03:03:06.873521 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.873525 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.873574 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.873676 (+ 102us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:06.874232 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.874803 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.875847 (+ 1044us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.876365 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.876409 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.876467 (+ 58us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.876535 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260131 03:03:06.877333 14864 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0237d95169314c52a6d915a736de4404" candidate_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6110ec810fd74ff2adf98a640f84b2e6" is_pre_election: true
I20260131 03:03:06.877333 14722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0237d95169314c52a6d915a736de4404" candidate_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6de748b72ca40209a51efb372bf1cfb" is_pre_election: true
I20260131 03:03:06.877632 14722 raft_consensus.cc:2468] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e846a385ec4544ad7ecde4a1336cf8 in term 0.
I20260131 03:03:06.877640 14864 raft_consensus.cc:2468] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e846a385ec4544ad7ecde4a1336cf8 in term 0.
I20260131 03:03:06.878116 14513 leader_election.cc:304] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6de748b72ca40209a51efb372bf1cfb, f8e846a385ec4544ad7ecde4a1336cf8; no voters:
I20260131 03:03:06.878479 14945 raft_consensus.cc:2804] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:03:06.878597 14945 raft_consensus.cc:493] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:03:06.878630 14945 raft_consensus.cc:3060] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:06.879842 14945 raft_consensus.cc:515] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } }
I20260131 03:03:06.880311 14945 leader_election.cc:290] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [CANDIDATE]: Term 1 election: Requested vote from peers 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951), e6de748b72ca40209a51efb372bf1cfb (127.6.22.66:41925)
I20260131 03:03:06.880977 14722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0237d95169314c52a6d915a736de4404" candidate_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6de748b72ca40209a51efb372bf1cfb"
I20260131 03:03:06.880993 14864 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0237d95169314c52a6d915a736de4404" candidate_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6110ec810fd74ff2adf98a640f84b2e6"
I20260131 03:03:06.881167 14722 raft_consensus.cc:3060] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:06.881184 14864 raft_consensus.cc:3060] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:06.882328 14722 raft_consensus.cc:2468] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e846a385ec4544ad7ecde4a1336cf8 in term 1.
I20260131 03:03:06.882335 14864 raft_consensus.cc:2468] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e846a385ec4544ad7ecde4a1336cf8 in term 1.
I20260131 03:03:06.882808 14513 leader_election.cc:304] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6de748b72ca40209a51efb372bf1cfb, f8e846a385ec4544ad7ecde4a1336cf8; no voters:
I20260131 03:03:06.883044 14945 raft_consensus.cc:2804] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:03:06.883314 14945 raft_consensus.cc:697] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 LEADER]: Becoming Leader. State: Replica: f8e846a385ec4544ad7ecde4a1336cf8, State: Running, Role: LEADER
I20260131 03:03:06.883801 14945 consensus_queue.cc:237] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } }
I20260131 03:03:06.888119 14421 catalog_manager.cc:5654] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65). New cstate: current_term: 1 leader_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } health_report { overall_health: UNKNOWN } } }
W20260131 03:03:06.899626 14627 tablet.cc:2378] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:03:06.909457 14911 tablet.cc:2378] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:03:06.912132 14769 tablet.cc:2378] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:03:06.912684 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:06.909320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.6.22.65:44813)
0131 03:03:06.909504 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:06.909509 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:06.909525 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:03:06.909686 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:06.909689 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:06.909770 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:06.909881 (+ 111us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:03:06.910220 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.910711 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:06.911376 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:06.911573 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:06.911627 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:03:06.912202 (+ 575us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:03:06.912337 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:06.912433 (+ 96us) server_negotiation.cc:300] Negotiation successful
0131 03:03:06.912504 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260131 03:03:06.918942 14864 raft_consensus.cc:1275] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Refusing update from remote peer f8e846a385ec4544ad7ecde4a1336cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:06.918943 14722 raft_consensus.cc:1275] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Refusing update from remote peer f8e846a385ec4544ad7ecde4a1336cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:06.919811 14949 consensus_queue.cc:1048] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:06.920091 14945 consensus_queue.cc:1048] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:06.929440 14954 mvcc.cc:204] Tried to move back new op lower bound from 7249217892010336256 to 7249217891881684992. Current Snapshot: MvccSnapshot[applied={T|T < 7249217892010336256}]
I20260131 03:03:06.930337 14953 mvcc.cc:204] Tried to move back new op lower bound from 7249217892010336256 to 7249217891881684992. Current Snapshot: MvccSnapshot[applied={T|T < 7249217892010336256}]
I20260131 03:03:06.938624 14418 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:03:06.938838 14418 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:03:06.941655 14418 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d: Sending DeleteTablet for 3 replicas of tablet 0237d95169314c52a6d915a736de4404
I20260131 03:03:06.942634 14558 tablet_service.cc:1552] Processing DeleteTablet for tablet 0237d95169314c52a6d915a736de4404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:06 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:57384
I20260131 03:03:06.942636 14844 tablet_service.cc:1552] Processing DeleteTablet for tablet 0237d95169314c52a6d915a736de4404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:06 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:38724
I20260131 03:03:06.942669 14702 tablet_service.cc:1552] Processing DeleteTablet for tablet 0237d95169314c52a6d915a736de4404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:06 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:49936
I20260131 03:03:06.943367 14961 tablet_replica.cc:333] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8: stopping tablet replica
I20260131 03:03:06.943671 14961 raft_consensus.cc:2243] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:03:06.944167 14961 raft_consensus.cc:2272] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:06.944082 14418 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260131 03:03:06.944550 14962 tablet_replica.cc:333] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6: stopping tablet replica
W20260131 03:03:06.944572 14418 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:03:06.944859 14962 raft_consensus.cc:2243] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:03:06.945212 14962 raft_consensus.cc:2272] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:06.946641 14961 ts_tablet_manager.cc:1916] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:06.949489 14961 ts_tablet_manager.cc:1929] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:03:06.949622 14961 log.cc:1199] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/wal/wals/0237d95169314c52a6d915a736de4404
I20260131 03:03:06.949934 14961 ts_tablet_manager.cc:1950] T 0237d95169314c52a6d915a736de4404 P f8e846a385ec4544ad7ecde4a1336cf8: Deleting consensus metadata
I20260131 03:03:06.951112 14558 tablet_service.cc:1505] Processing CreateTablet for tablet d8a40bbc03bd41acb5809d76e0243790 (DEFAULT_TABLE table=test-table [id=08ab9a3cd8934d9ea3042269e8e8a1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:06.951408 14558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8a40bbc03bd41acb5809d76e0243790. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:06.951555 14844 tablet_service.cc:1505] Processing CreateTablet for tablet d8a40bbc03bd41acb5809d76e0243790 (DEFAULT_TABLE table=test-table [id=08ab9a3cd8934d9ea3042269e8e8a1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:06.951812 14844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8a40bbc03bd41acb5809d76e0243790. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:06.951920 14962 ts_tablet_manager.cc:1916] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:06.952891 14963 tablet_replica.cc:333] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb: stopping tablet replica
I20260131 03:03:06.953217 14963 raft_consensus.cc:2243] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:03:06.953652 14963 raft_consensus.cc:2272] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:06.954069 14962 ts_tablet_manager.cc:1929] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:03:06.954149 14962 log.cc:1199] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/0237d95169314c52a6d915a736de4404
I20260131 03:03:06.954151 14939 tablet_bootstrap.cc:492] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8: Bootstrap starting.
I20260131 03:03:06.954475 14962 ts_tablet_manager.cc:1950] T 0237d95169314c52a6d915a736de4404 P 6110ec810fd74ff2adf98a640f84b2e6: Deleting consensus metadata
I20260131 03:03:06.954530 14702 tablet_service.cc:1505] Processing CreateTablet for tablet d8a40bbc03bd41acb5809d76e0243790 (DEFAULT_TABLE table=test-table [id=08ab9a3cd8934d9ea3042269e8e8a1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:06.954818 14702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8a40bbc03bd41acb5809d76e0243790. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:06.955291 14939 tablet_bootstrap.cc:654] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:06.956245 14407 catalog_manager.cc:4985] TS 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951): tablet 0237d95169314c52a6d915a736de4404 (table test-table [id=2eeaf8bd9a6d427c8d7ed500787f42eb]) successfully deleted
I20260131 03:03:06.956523 14407 catalog_manager.cc:4985] TS f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65:44813): tablet 0237d95169314c52a6d915a736de4404 (table test-table [id=2eeaf8bd9a6d427c8d7ed500787f42eb]) successfully deleted
I20260131 03:03:06.956602 14940 tablet_bootstrap.cc:492] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6: Bootstrap starting.
I20260131 03:03:06.957165 14938 tablet_bootstrap.cc:492] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb: Bootstrap starting.
I20260131 03:03:06.957446 14939 tablet_bootstrap.cc:492] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8: No bootstrap required, opened a new log
I20260131 03:03:06.957526 14939 ts_tablet_manager.cc:1403] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260131 03:03:06.958055 14940 tablet_bootstrap.cc:654] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:06.958087 14939 raft_consensus.cc:359] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:06.958175 14938 tablet_bootstrap.cc:654] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:06.958209 14939 raft_consensus.cc:385] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:06.958240 14939 raft_consensus.cc:740] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e846a385ec4544ad7ecde4a1336cf8, State: Initialized, Role: FOLLOWER
I20260131 03:03:06.958560 14939 consensus_queue.cc:260] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:06.959072 14939 ts_tablet_manager.cc:1434] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:03:06.959666 14938 tablet_bootstrap.cc:492] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb: No bootstrap required, opened a new log
I20260131 03:03:06.959735 14940 tablet_bootstrap.cc:492] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6: No bootstrap required, opened a new log
I20260131 03:03:06.959753 14938 ts_tablet_manager.cc:1403] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260131 03:03:06.959827 14940 ts_tablet_manager.cc:1403] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260131 03:03:06.960206 14938 raft_consensus.cc:359] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:06.960305 14938 raft_consensus.cc:385] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:06.960330 14963 ts_tablet_manager.cc:1916] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:06.960335 14938 raft_consensus.cc:740] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6de748b72ca40209a51efb372bf1cfb, State: Initialized, Role: FOLLOWER
I20260131 03:03:06.960398 14940 raft_consensus.cc:359] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:06.960516 14940 raft_consensus.cc:385] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:06.960583 14940 raft_consensus.cc:740] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6110ec810fd74ff2adf98a640f84b2e6, State: Initialized, Role: FOLLOWER
I20260131 03:03:06.960593 14938 consensus_queue.cc:260] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:06.960772 14940 consensus_queue.cc:260] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:06.960922 14938 ts_tablet_manager.cc:1434] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:03:06.961087 14940 ts_tablet_manager.cc:1434] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260131 03:03:06.963117 14963 ts_tablet_manager.cc:1929] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:03:06.963228 14963 log.cc:1199] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/0237d95169314c52a6d915a736de4404
I20260131 03:03:06.963555 14963 ts_tablet_manager.cc:1950] T 0237d95169314c52a6d915a736de4404 P e6de748b72ca40209a51efb372bf1cfb: Deleting consensus metadata
I20260131 03:03:06.964746 14407 catalog_manager.cc:4985] TS e6de748b72ca40209a51efb372bf1cfb (127.6.22.66:41925): tablet 0237d95169314c52a6d915a736de4404 (table test-table [id=2eeaf8bd9a6d427c8d7ed500787f42eb]) successfully deleted
I20260131 03:03:07.042567 14946 raft_consensus.cc:493] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:03:07.042757 14946 raft_consensus.cc:515] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.044131 14946 leader_election.cc:290] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65:44813), 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951)
I20260131 03:03:07.048339 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.044410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:44813 (local address 127.6.22.66:50061)
0131 03:03:07.044724 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:07.044743 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:07.044908 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:07.045332 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:07.045337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:07.045358 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:07.046011 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.046027 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.046847 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.046853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:07.047863 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.047874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.048002 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.048032 (+ 30us) client_negotiation.cc:769] Sending connection context
0131 03:03:07.048083 (+ 51us) client_negotiation.cc:240] Negotiation successful
0131 03:03:07.048147 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":102,"threads_started":1}
I20260131 03:03:07.048867 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.044481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:50061 (local address 127.6.22.65:44813)
0131 03:03:07.044682 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.044689 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.044815 (+ 126us) server_negotiation.cc:408] Connection header received
0131 03:03:07.045064 (+ 249us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.045070 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.045129 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.045182 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:07.046184 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.046717 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.048013 (+ 1296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.048497 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.048541 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.048600 (+ 59us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.048729 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260131 03:03:07.049032 14935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.044973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:35823 (local address 127.6.22.67:33951)
0131 03:03:07.045113 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.045118 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.045278 (+ 160us) server_negotiation.cc:408] Connection header received
0131 03:03:07.045383 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.045387 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.045448 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.045555 (+ 107us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:07.046092 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.046852 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.048233 (+ 1381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.048728 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.048769 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.048831 (+ 62us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.048900 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260131 03:03:07.048877 14968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.044845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:33951 (local address 127.6.22.66:35823)
0131 03:03:07.045185 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:07.045198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:07.045286 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:07.045579 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:07.045581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:07.045600 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:07.045938 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.045956 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.046968 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.046971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:07.047863 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.047874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.048003 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.048025 (+ 22us) client_negotiation.cc:769] Sending connection context
0131 03:03:07.048083 (+ 58us) client_negotiation.cc:240] Negotiation successful
0131 03:03:07.048147 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260131 03:03:07.049701 14864 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8a40bbc03bd41acb5809d76e0243790" candidate_uuid: "e6de748b72ca40209a51efb372bf1cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6110ec810fd74ff2adf98a640f84b2e6" is_pre_election: true
I20260131 03:03:07.049878 14864 raft_consensus.cc:2468] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6de748b72ca40209a51efb372bf1cfb in term 0.
I20260131 03:03:07.049875 14580 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8a40bbc03bd41acb5809d76e0243790" candidate_uuid: "e6de748b72ca40209a51efb372bf1cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" is_pre_election: true
I20260131 03:03:07.050103 14580 raft_consensus.cc:2468] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6de748b72ca40209a51efb372bf1cfb in term 0.
I20260131 03:03:07.050297 14655 leader_election.cc:304] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6110ec810fd74ff2adf98a640f84b2e6, e6de748b72ca40209a51efb372bf1cfb; no voters:
I20260131 03:03:07.050570 14946 raft_consensus.cc:2804] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:03:07.050654 14946 raft_consensus.cc:493] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:03:07.050704 14946 raft_consensus.cc:3060] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:07.051620 14946 raft_consensus.cc:515] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.052004 14946 leader_election.cc:290] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [CANDIDATE]: Term 1 election: Requested vote from peers f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65:44813), 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951)
I20260131 03:03:07.052520 14864 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8a40bbc03bd41acb5809d76e0243790" candidate_uuid: "e6de748b72ca40209a51efb372bf1cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6110ec810fd74ff2adf98a640f84b2e6"
I20260131 03:03:07.052531 14580 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8a40bbc03bd41acb5809d76e0243790" candidate_uuid: "e6de748b72ca40209a51efb372bf1cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e846a385ec4544ad7ecde4a1336cf8"
I20260131 03:03:07.052636 14864 raft_consensus.cc:3060] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:07.052703 14580 raft_consensus.cc:3060] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:07.053628 14864 raft_consensus.cc:2468] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6de748b72ca40209a51efb372bf1cfb in term 1.
I20260131 03:03:07.053927 14580 raft_consensus.cc:2468] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6de748b72ca40209a51efb372bf1cfb in term 1.
I20260131 03:03:07.053973 14655 leader_election.cc:304] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6110ec810fd74ff2adf98a640f84b2e6, e6de748b72ca40209a51efb372bf1cfb; no voters:
I20260131 03:03:07.054205 14946 raft_consensus.cc:2804] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:03:07.054477 14946 raft_consensus.cc:697] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 1 LEADER]: Becoming Leader. State: Replica: e6de748b72ca40209a51efb372bf1cfb, State: Running, Role: LEADER
I20260131 03:03:07.054768 14946 consensus_queue.cc:237] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.057483 14421 catalog_manager.cc:5654] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb reported cstate change: term changed from 0 to 1, leader changed from <none> to e6de748b72ca40209a51efb372bf1cfb (127.6.22.66). New cstate: current_term: 1 leader_uuid: "e6de748b72ca40209a51efb372bf1cfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } health_report { overall_health: UNKNOWN } } }
Jan 31 03:03:07 dist-test-slave-j368 krb5kdc[14373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828585, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.126@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:03:07.119310 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.110525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:46517 (local address 127.0.0.1:51632)
0131 03:03:07.110879 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:07.110897 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:07.111041 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:07.111320 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:07.111324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:07.111883 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:07.112160 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.112170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.112989 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.112993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:07.113575 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.113586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.113711 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.114320 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:07.114339 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:07.115914 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:07.117789 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:07.117798 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:07.117804 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:07.118023 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:07.118280 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:07.118285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:07.118287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:07.118334 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:07.118712 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:07.118719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:07.118814 (+ 95us) client_negotiation.cc:769] Sending connection context
0131 03:03:07.118928 (+ 114us) client_negotiation.cc:240] Negotiation successful
0131 03:03:07.119084 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":132,"threads_started":1}
I20260131 03:03:07.119310 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.110664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.6.22.126:46517)
0131 03:03:07.110797 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.110801 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.110988 (+ 187us) server_negotiation.cc:408] Connection header received
0131 03:03:07.111139 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.111142 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.111182 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.111253 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:07.112289 (+ 1036us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.112866 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.113705 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.113827 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.116049 (+ 2222us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:07.116074 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:07.116076 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:07.116098 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:07.117626 (+ 1528us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:07.118137 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:07.118140 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:07.118142 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:07.118189 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:07.118422 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:07.118426 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:07.118428 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:07.118580 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:07.118688 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.118992 (+ 304us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.119170 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":54}
I20260131 03:03:07.123327 14421 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:51632:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260131 03:03:07.124027 14421 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:03:07.129182 14558 tablet_service.cc:1505] Processing CreateTablet for tablet 117713a2444d4f7d9cb601388a1d8316 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a84e7f983eba49a1926dc56f1a47f4b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:03:07.129568 14558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 117713a2444d4f7d9cb601388a1d8316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:07.129472 14702 tablet_service.cc:1505] Processing CreateTablet for tablet 117713a2444d4f7d9cb601388a1d8316 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a84e7f983eba49a1926dc56f1a47f4b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:03:07.129832 14702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 117713a2444d4f7d9cb601388a1d8316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:07.131789 14939 tablet_bootstrap.cc:492] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8: Bootstrap starting.
I20260131 03:03:07.132287 14844 tablet_service.cc:1505] Processing CreateTablet for tablet 117713a2444d4f7d9cb601388a1d8316 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a84e7f983eba49a1926dc56f1a47f4b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:03:07.132634 14844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 117713a2444d4f7d9cb601388a1d8316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:07.132797 14938 tablet_bootstrap.cc:492] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb: Bootstrap starting.
I20260131 03:03:07.132959 14939 tablet_bootstrap.cc:654] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:07.133744 14938 tablet_bootstrap.cc:654] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:07.135030 14938 tablet_bootstrap.cc:492] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb: No bootstrap required, opened a new log
I20260131 03:03:07.135113 14938 ts_tablet_manager.cc:1403] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:03:07.135548 14938 raft_consensus.cc:359] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.135670 14938 raft_consensus.cc:385] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:07.135708 14938 raft_consensus.cc:740] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6de748b72ca40209a51efb372bf1cfb, State: Initialized, Role: FOLLOWER
I20260131 03:03:07.135834 14938 consensus_queue.cc:260] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.136104 14938 ts_tablet_manager.cc:1434] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:03:07.136297 14940 tablet_bootstrap.cc:492] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: Bootstrap starting.
I20260131 03:03:07.136518 14939 tablet_bootstrap.cc:492] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8: No bootstrap required, opened a new log
I20260131 03:03:07.136627 14939 ts_tablet_manager.cc:1403] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260131 03:03:07.137101 14939 raft_consensus.cc:359] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.137229 14939 raft_consensus.cc:385] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:07.137270 14939 raft_consensus.cc:740] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e846a385ec4544ad7ecde4a1336cf8, State: Initialized, Role: FOLLOWER
I20260131 03:03:07.137336 14940 tablet_bootstrap.cc:654] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:07.137423 14939 consensus_queue.cc:260] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.137729 14939 ts_tablet_manager.cc:1434] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260131 03:03:07.137962 14949 tablet_replica.cc:442] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } }
I20260131 03:03:07.138080 14949 tablet_replica.cc:445] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:03:07.138356 14946 tablet_replica.cc:442] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } }
I20260131 03:03:07.138471 14946 tablet_replica.cc:445] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:03:07.141273 14940 tablet_bootstrap.cc:492] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: No bootstrap required, opened a new log
I20260131 03:03:07.141386 14940 ts_tablet_manager.cc:1403] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260131 03:03:07.141976 14940 raft_consensus.cc:359] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.142114 14940 raft_consensus.cc:385] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:07.142153 14940 raft_consensus.cc:740] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6110ec810fd74ff2adf98a640f84b2e6, State: Initialized, Role: FOLLOWER
I20260131 03:03:07.142305 14940 consensus_queue.cc:260] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.142591 14944 tablet_replica.cc:442] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } }
I20260131 03:03:07.142731 14944 tablet_replica.cc:445] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:03:07.142611 14940 ts_tablet_manager.cc:1434] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260131 03:03:07.152710 14944 raft_consensus.cc:493] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:03:07.152930 14944 raft_consensus.cc:515] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.154744 14944 leader_election.cc:290] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65:44813), e6de748b72ca40209a51efb372bf1cfb (127.6.22.66:41925)
I20260131 03:03:07.158960 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.154980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:44813 (local address 127.6.22.67:60905)
0131 03:03:07.155183 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:07.155213 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:07.155400 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:07.156144 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:07.156148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:07.156173 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:07.156425 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.156431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.157580 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.157584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:07.158521 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.158531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.158653 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.158680 (+ 27us) client_negotiation.cc:769] Sending connection context
0131 03:03:07.158731 (+ 51us) client_negotiation.cc:240] Negotiation successful
0131 03:03:07.158816 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260131 03:03:07.160034 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.155678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:60905 (local address 127.6.22.65:44813)
0131 03:03:07.155794 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.155798 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.155814 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:03:07.155869 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.155872 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.155920 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.156015 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:07.156551 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.157419 (+ 868us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.159282 (+ 1863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.159765 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.159804 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.159851 (+ 47us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.159913 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260131 03:03:07.160405 14937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.155384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:55861 (local address 127.6.22.66:41925)
0131 03:03:07.155518 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.155523 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.155795 (+ 272us) server_negotiation.cc:408] Connection header received
0131 03:03:07.156024 (+ 229us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.156028 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.156092 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.156148 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:07.156912 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.157751 (+ 839us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.159386 (+ 1635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.159888 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.159939 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.160177 (+ 238us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.160260 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260131 03:03:07.160449 14580 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "117713a2444d4f7d9cb601388a1d8316" candidate_uuid: "6110ec810fd74ff2adf98a640f84b2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" is_pre_election: true
I20260131 03:03:07.160605 14580 raft_consensus.cc:2468] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6110ec810fd74ff2adf98a640f84b2e6 in term 0.
I20260131 03:03:07.160897 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.155311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:41925 (local address 127.6.22.67:55861)
0131 03:03:07.155685 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:07.155701 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:07.155879 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:07.156323 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:07.156325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:07.156343 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:07.156750 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.156760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.157910 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.157914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:07.159247 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.159257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.160005 (+ 748us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.160028 (+ 23us) client_negotiation.cc:769] Sending connection context
0131 03:03:07.160645 (+ 617us) client_negotiation.cc:240] Negotiation successful
0131 03:03:07.160744 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":110,"threads_started":1}
I20260131 03:03:07.161089 14797 leader_election.cc:304] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6110ec810fd74ff2adf98a640f84b2e6, f8e846a385ec4544ad7ecde4a1336cf8; no voters:
I20260131 03:03:07.161401 14944 raft_consensus.cc:2804] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:03:07.161477 14944 raft_consensus.cc:493] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:03:07.161523 14944 raft_consensus.cc:3060] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:07.162870 14944 raft_consensus.cc:515] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.163369 14944 leader_election.cc:290] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [CANDIDATE]: Term 1 election: Requested vote from peers f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65:44813), e6de748b72ca40209a51efb372bf1cfb (127.6.22.66:41925)
I20260131 03:03:07.163800 14580 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "117713a2444d4f7d9cb601388a1d8316" candidate_uuid: "6110ec810fd74ff2adf98a640f84b2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e846a385ec4544ad7ecde4a1336cf8"
I20260131 03:03:07.163949 14580 raft_consensus.cc:3060] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:07.163902 14722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "117713a2444d4f7d9cb601388a1d8316" candidate_uuid: "6110ec810fd74ff2adf98a640f84b2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6de748b72ca40209a51efb372bf1cfb" is_pre_election: true
I20260131 03:03:07.164057 14722 raft_consensus.cc:2468] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6110ec810fd74ff2adf98a640f84b2e6 in term 0.
I20260131 03:03:07.164287 14721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "117713a2444d4f7d9cb601388a1d8316" candidate_uuid: "6110ec810fd74ff2adf98a640f84b2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6de748b72ca40209a51efb372bf1cfb"
I20260131 03:03:07.164371 14721 raft_consensus.cc:3060] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:07.165143 14580 raft_consensus.cc:2468] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6110ec810fd74ff2adf98a640f84b2e6 in term 1.
I20260131 03:03:07.165491 14797 leader_election.cc:304] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6110ec810fd74ff2adf98a640f84b2e6, f8e846a385ec4544ad7ecde4a1336cf8; no voters:
I20260131 03:03:07.165630 14721 raft_consensus.cc:2468] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6110ec810fd74ff2adf98a640f84b2e6 in term 1.
I20260131 03:03:07.166251 14944 raft_consensus.cc:2804] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:03:07.166574 14944 raft_consensus.cc:697] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 LEADER]: Becoming Leader. State: Replica: 6110ec810fd74ff2adf98a640f84b2e6, State: Running, Role: LEADER
I20260131 03:03:07.166898 14944 consensus_queue.cc:237] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.169334 14981 tablet_replica.cc:442] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: TxnStatusTablet state changed. Reason: New leader 6110ec810fd74ff2adf98a640f84b2e6. Latest consensus state: current_term: 1 leader_uuid: "6110ec810fd74ff2adf98a640f84b2e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } }
I20260131 03:03:07.169440 14981 tablet_replica.cc:445] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:03:07.169907 14418 catalog_manager.cc:5654] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67). New cstate: current_term: 1 leader_uuid: "6110ec810fd74ff2adf98a640f84b2e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } health_report { overall_health: HEALTHY } } }
I20260131 03:03:07.171855 14983 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260131 03:03:07.172065 14983 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260131 03:03:07.185307 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.181056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:33951 (local address 127.0.0.1:38730)
0131 03:03:07.181222 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:07.181241 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:07.181362 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:07.181767 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:07.181769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:07.181776 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:03:07.181972 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.181977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.183049 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.183054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:07.183690 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.183703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.183863 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.183933 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.184983 (+ 1050us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.185006 (+ 23us) client_negotiation.cc:769] Sending connection context
0131 03:03:07.185077 (+ 71us) client_negotiation.cc:240] Negotiation successful
0131 03:03:07.185179 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260131 03:03:07.185326 14935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.181311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.6.22.67:33951)
0131 03:03:07.181472 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.181477 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.181494 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:03:07.181571 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.181575 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.181646 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.181768 (+ 122us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:03:07.182085 (+ 317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.182841 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.183870 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.184086 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.184162 (+ 76us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.184839 (+ 677us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.184966 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.185120 (+ 154us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.185181 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260131 03:03:07.188905 14721 raft_consensus.cc:1275] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Refusing update from remote peer 6110ec810fd74ff2adf98a640f84b2e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:07.189080 14580 raft_consensus.cc:1275] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Refusing update from remote peer 6110ec810fd74ff2adf98a640f84b2e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:07.189612 14944 consensus_queue.cc:1048] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:07.189822 14981 consensus_queue.cc:1048] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:07.192518 14949 tablet_replica.cc:442] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8: TxnStatusTablet state changed. Reason: New leader 6110ec810fd74ff2adf98a640f84b2e6. Latest consensus state: current_term: 1 leader_uuid: "6110ec810fd74ff2adf98a640f84b2e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } }
I20260131 03:03:07.192638 14949 tablet_replica.cc:445] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:03:07.193125 14981 tablet_replica.cc:442] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6110ec810fd74ff2adf98a640f84b2e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } }
I20260131 03:03:07.193231 14981 tablet_replica.cc:445] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:03:07.193631 14946 tablet_replica.cc:442] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb: TxnStatusTablet state changed. Reason: New leader 6110ec810fd74ff2adf98a640f84b2e6. Latest consensus state: current_term: 1 leader_uuid: "6110ec810fd74ff2adf98a640f84b2e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } }
I20260131 03:03:07.193738 14946 tablet_replica.cc:445] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:03:07.195319 14946 tablet_replica.cc:442] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6110ec810fd74ff2adf98a640f84b2e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } }
I20260131 03:03:07.195425 14946 tablet_replica.cc:445] T 117713a2444d4f7d9cb601388a1d8316 P e6de748b72ca40209a51efb372bf1cfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:03:07.196208 14944 tablet_replica.cc:442] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6110ec810fd74ff2adf98a640f84b2e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } }
I20260131 03:03:07.196313 14944 tablet_replica.cc:445] T 117713a2444d4f7d9cb601388a1d8316 P 6110ec810fd74ff2adf98a640f84b2e6: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:03:07.197633 14952 mvcc.cc:204] Tried to move back new op lower bound from 7249217893119954944 to 7249217893039665152. Current Snapshot: MvccSnapshot[applied={T|T < 7249217893119954944}]
I20260131 03:03:07.197829 14949 tablet_replica.cc:442] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6110ec810fd74ff2adf98a640f84b2e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } } }
I20260131 03:03:07.197925 14949 tablet_replica.cc:445] T 117713a2444d4f7d9cb601388a1d8316 P f8e846a385ec4544ad7ecde4a1336cf8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:03:07.205324 14937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.201306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.6.22.66:41925)
0131 03:03:07.201462 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.201467 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.201482 (+ 15us) server_negotiation.cc:408] Connection header received
0131 03:03:07.201577 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.201580 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.201650 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.201769 (+ 119us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:03:07.202094 (+ 325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.202915 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.203566 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.203775 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.203845 (+ 70us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.204500 (+ 655us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.204629 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.205016 (+ 387us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.205103 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260131 03:03:07.217814 14935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.214269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:33145 (local address 127.6.22.67:33951)
0131 03:03:07.214406 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.214412 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.214698 (+ 286us) server_negotiation.cc:408] Connection header received
0131 03:03:07.214956 (+ 258us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.214959 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.215012 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.215091 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:03:07.215575 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.216141 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.217177 (+ 1036us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.217316 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.217358 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.217432 (+ 74us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.217499 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.217650 (+ 151us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.217697 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260131 03:03:07.217883 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.214100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:33951 (local address 127.6.22.66:33145)
0131 03:03:07.214590 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:07.214605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:07.214738 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:07.215132 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:07.215136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:07.215145 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:03:07.215461 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.215468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.216280 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.216284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:07.217041 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.217052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.217193 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.217253 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.217532 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.217538 (+ 6us) client_negotiation.cc:769] Sending connection context
0131 03:03:07.217629 (+ 91us) client_negotiation.cc:240] Negotiation successful
0131 03:03:07.217707 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":157,"threads_started":1}
I20260131 03:03:07.224990 14580 raft_consensus.cc:1275] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Refusing update from remote peer e6de748b72ca40209a51efb372bf1cfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:07.224990 14864 raft_consensus.cc:1275] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Refusing update from remote peer e6de748b72ca40209a51efb372bf1cfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:07.225677 14969 consensus_queue.cc:1048] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:07.226087 14946 consensus_queue.cc:1048] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:07.250581 14937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.247070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:32837 (local address 127.6.22.66:41925)
0131 03:03:07.247208 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.247212 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.247630 (+ 418us) server_negotiation.cc:408] Connection header received
0131 03:03:07.247756 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.247759 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.247808 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.247874 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:03:07.248387 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.248948 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.249825 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.250040 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.250090 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.250183 (+ 93us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.250265 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.250384 (+ 119us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.250437 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260131 03:03:07.250594 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.246745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:41925 (local address 127.6.22.67:32837)
0131 03:03:07.247545 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:07.247560 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:07.247671 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:07.247918 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:07.247921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:07.247941 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:03:07.248276 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.248284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.249096 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.249099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:07.249691 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.249701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.249835 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.249892 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.250282 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.250286 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:03:07.250367 (+ 81us) client_negotiation.cc:240] Negotiation successful
0131 03:03:07.250445 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":122,"threads_started":1}
I20260131 03:03:07.279505 14420 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:03:07.279706 14420 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:03:07.282397 14420 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d: Sending DeleteTablet for 3 replicas of tablet d8a40bbc03bd41acb5809d76e0243790
I20260131 03:03:07.283424 14844 tablet_service.cc:1552] Processing DeleteTablet for tablet d8a40bbc03bd41acb5809d76e0243790 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:07 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:38724
I20260131 03:03:07.283435 14558 tablet_service.cc:1552] Processing DeleteTablet for tablet d8a40bbc03bd41acb5809d76e0243790 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:07 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:57384
I20260131 03:03:07.283651 14961 tablet_replica.cc:333] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8: stopping tablet replica
I20260131 03:03:07.283788 14961 raft_consensus.cc:2243] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:03:07.283978 14961 raft_consensus.cc:2272] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:07.284178 14962 tablet_replica.cc:333] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6: stopping tablet replica
I20260131 03:03:07.284305 14962 raft_consensus.cc:2243] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:03:07.284425 14962 raft_consensus.cc:2272] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:07.284931 14702 tablet_service.cc:1552] Processing DeleteTablet for tablet d8a40bbc03bd41acb5809d76e0243790 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:07 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:49936
I20260131 03:03:07.285128 14963 tablet_replica.cc:333] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb: stopping tablet replica
I20260131 03:03:07.285239 14963 raft_consensus.cc:2243] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:03:07.285557 14963 raft_consensus.cc:2272] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:07.286587 14961 ts_tablet_manager.cc:1916] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:07.289595 14961 ts_tablet_manager.cc:1929] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:03:07.289696 14961 log.cc:1199] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/wal/wals/d8a40bbc03bd41acb5809d76e0243790
I20260131 03:03:07.289985 14961 ts_tablet_manager.cc:1950] T d8a40bbc03bd41acb5809d76e0243790 P f8e846a385ec4544ad7ecde4a1336cf8: Deleting consensus metadata
I20260131 03:03:07.290855 14407 catalog_manager.cc:4985] TS f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65:44813): tablet d8a40bbc03bd41acb5809d76e0243790 (table test-table [id=08ab9a3cd8934d9ea3042269e8e8a1c3]) successfully deleted
I20260131 03:03:07.291776 14963 ts_tablet_manager.cc:1916] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:07.294643 14963 ts_tablet_manager.cc:1929] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:03:07.295738 14962 ts_tablet_manager.cc:1916] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:07.298683 14962 ts_tablet_manager.cc:1929] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:03:07.298791 14962 log.cc:1199] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/d8a40bbc03bd41acb5809d76e0243790
I20260131 03:03:07.299070 14962 ts_tablet_manager.cc:1950] T d8a40bbc03bd41acb5809d76e0243790 P 6110ec810fd74ff2adf98a640f84b2e6: Deleting consensus metadata
Jan 31 03:03:07 dist-test-slave-j368 krb5kdc[14373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260131 03:03:07.299942 14407 catalog_manager.cc:4985] TS 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951): tablet d8a40bbc03bd41acb5809d76e0243790 (table test-table [id=08ab9a3cd8934d9ea3042269e8e8a1c3]) successfully deleted
I20260131 03:03:07.294741 14963 log.cc:1199] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/d8a40bbc03bd41acb5809d76e0243790
I20260131 03:03:07.300457 14963 ts_tablet_manager.cc:1950] T d8a40bbc03bd41acb5809d76e0243790 P e6de748b72ca40209a51efb372bf1cfb: Deleting consensus metadata
I20260131 03:03:07.301270 14407 catalog_manager.cc:4985] TS e6de748b72ca40209a51efb372bf1cfb (127.6.22.66:41925): tablet d8a40bbc03bd41acb5809d76e0243790 (table test-table [id=08ab9a3cd8934d9ea3042269e8e8a1c3]) successfully deleted
Jan 31 03:03:07 dist-test-slave-j368 krb5kdc[14373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:07 dist-test-slave-j368 krb5kdc[14373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
W20260131 03:03:07.317152 14470 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:07.309367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.6.22.126:46517)
0131 03:03:07.309508 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.309513 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.309791 (+ 278us) server_negotiation.cc:408] Connection header received
0131 03:03:07.309915 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.309919 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.309981 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.310074 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:07.311190 (+ 1116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.312070 (+ 880us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.313138 (+ 1068us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.313341 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.317004 (+ 3663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51634: BlockingRecv error: recv got EOF from 127.0.0.1:51634 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jan 31 03:03:07 dist-test-slave-j368 krb5kdc[14373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM
I20260131 03:03:07.329752 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.319203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.6.22.126:46517)
0131 03:03:07.319335 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.319340 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.319807 (+ 467us) server_negotiation.cc:408] Connection header received
0131 03:03:07.319941 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.319945 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.320005 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.320101 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:07.321256 (+ 1155us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.322193 (+ 937us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.323095 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.323294 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.326064 (+ 2770us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:07.326091 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:07.326095 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:07.326137 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:07.327919 (+ 1782us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:07.328440 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:07.328444 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:07.328446 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:07.328501 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:07.328779 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:07.328783 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:07.328785 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:07.329009 (+ 224us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:07.329213 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.329442 (+ 229us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.329572 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260131 03:03:07.333055 14420 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:03:07.333561 14420 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:03:07.338984 14558 tablet_service.cc:1505] Processing CreateTablet for tablet 8a3fceae8dc04affa0b9b37111228cfb (DEFAULT_TABLE table=test-table [id=a5fc64c0d1244c28acb5cc242736fcec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:07.339304 14558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a3fceae8dc04affa0b9b37111228cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:07.339464 14702 tablet_service.cc:1505] Processing CreateTablet for tablet 8a3fceae8dc04affa0b9b37111228cfb (DEFAULT_TABLE table=test-table [id=a5fc64c0d1244c28acb5cc242736fcec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:07.339725 14702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a3fceae8dc04affa0b9b37111228cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:07.341624 14939 tablet_bootstrap.cc:492] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8: Bootstrap starting.
I20260131 03:03:07.342192 14844 tablet_service.cc:1505] Processing CreateTablet for tablet 8a3fceae8dc04affa0b9b37111228cfb (DEFAULT_TABLE table=test-table [id=a5fc64c0d1244c28acb5cc242736fcec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:07.342448 14844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a3fceae8dc04affa0b9b37111228cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:07.342554 14939 tablet_bootstrap.cc:654] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:07.343626 14939 tablet_bootstrap.cc:492] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8: No bootstrap required, opened a new log
I20260131 03:03:07.343690 14939 ts_tablet_manager.cc:1403] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:03:07.344031 14939 raft_consensus.cc:359] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.344111 14939 raft_consensus.cc:385] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:07.344132 14939 raft_consensus.cc:740] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e846a385ec4544ad7ecde4a1336cf8, State: Initialized, Role: FOLLOWER
I20260131 03:03:07.344282 14939 consensus_queue.cc:260] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.344558 14939 ts_tablet_manager.cc:1434] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260131 03:03:07.344637 14938 tablet_bootstrap.cc:492] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb: Bootstrap starting.
I20260131 03:03:07.345666 14938 tablet_bootstrap.cc:654] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:07.346297 14940 tablet_bootstrap.cc:492] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6: Bootstrap starting.
I20260131 03:03:07.347133 14940 tablet_bootstrap.cc:654] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:07.349225 14938 tablet_bootstrap.cc:492] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb: No bootstrap required, opened a new log
I20260131 03:03:07.349327 14938 ts_tablet_manager.cc:1403] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260131 03:03:07.349817 14938 raft_consensus.cc:359] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.349943 14938 raft_consensus.cc:385] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:07.349988 14938 raft_consensus.cc:740] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6de748b72ca40209a51efb372bf1cfb, State: Initialized, Role: FOLLOWER
I20260131 03:03:07.350122 14938 consensus_queue.cc:260] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.350405 14938 ts_tablet_manager.cc:1434] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:03:07.350987 14940 tablet_bootstrap.cc:492] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6: No bootstrap required, opened a new log
I20260131 03:03:07.351074 14940 ts_tablet_manager.cc:1403] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260131 03:03:07.351537 14940 raft_consensus.cc:359] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.351642 14940 raft_consensus.cc:385] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:07.351676 14940 raft_consensus.cc:740] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6110ec810fd74ff2adf98a640f84b2e6, State: Initialized, Role: FOLLOWER
I20260131 03:03:07.351804 14940 consensus_queue.cc:260] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.352093 14940 ts_tablet_manager.cc:1434] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260131 03:03:07.492115 14969 raft_consensus.cc:493] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:03:07.492304 14969 raft_consensus.cc:515] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.492848 14969 leader_election.cc:290] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65:44813), 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951)
I20260131 03:03:07.493436 14580 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a3fceae8dc04affa0b9b37111228cfb" candidate_uuid: "e6de748b72ca40209a51efb372bf1cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" is_pre_election: true
I20260131 03:03:07.493479 14864 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a3fceae8dc04affa0b9b37111228cfb" candidate_uuid: "e6de748b72ca40209a51efb372bf1cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6110ec810fd74ff2adf98a640f84b2e6" is_pre_election: true
I20260131 03:03:07.493603 14864 raft_consensus.cc:2468] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6de748b72ca40209a51efb372bf1cfb in term 0.
I20260131 03:03:07.493603 14580 raft_consensus.cc:2468] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6de748b72ca40209a51efb372bf1cfb in term 0.
I20260131 03:03:07.493911 14655 leader_election.cc:304] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6110ec810fd74ff2adf98a640f84b2e6, e6de748b72ca40209a51efb372bf1cfb; no voters:
I20260131 03:03:07.494124 14969 raft_consensus.cc:2804] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:03:07.494194 14969 raft_consensus.cc:493] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:03:07.494230 14969 raft_consensus.cc:3060] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:07.495465 14969 raft_consensus.cc:515] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.495890 14969 leader_election.cc:290] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [CANDIDATE]: Term 1 election: Requested vote from peers f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65:44813), 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951)
I20260131 03:03:07.496284 14580 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a3fceae8dc04affa0b9b37111228cfb" candidate_uuid: "e6de748b72ca40209a51efb372bf1cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e846a385ec4544ad7ecde4a1336cf8"
I20260131 03:03:07.496415 14580 raft_consensus.cc:3060] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:07.496402 14864 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a3fceae8dc04affa0b9b37111228cfb" candidate_uuid: "e6de748b72ca40209a51efb372bf1cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6110ec810fd74ff2adf98a640f84b2e6"
I20260131 03:03:07.496538 14864 raft_consensus.cc:3060] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:07.497450 14864 raft_consensus.cc:2468] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6de748b72ca40209a51efb372bf1cfb in term 1.
I20260131 03:03:07.497488 14580 raft_consensus.cc:2468] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6de748b72ca40209a51efb372bf1cfb in term 1.
I20260131 03:03:07.497897 14655 leader_election.cc:304] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6de748b72ca40209a51efb372bf1cfb, f8e846a385ec4544ad7ecde4a1336cf8; no voters:
I20260131 03:03:07.498087 14969 raft_consensus.cc:2804] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:03:07.498176 14969 raft_consensus.cc:697] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 1 LEADER]: Becoming Leader. State: Replica: e6de748b72ca40209a51efb372bf1cfb, State: Running, Role: LEADER
I20260131 03:03:07.498325 14969 consensus_queue.cc:237] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.500420 14418 catalog_manager.cc:5654] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb reported cstate change: term changed from 0 to 1, leader changed from <none> to e6de748b72ca40209a51efb372bf1cfb (127.6.22.66). New cstate: current_term: 1 leader_uuid: "e6de748b72ca40209a51efb372bf1cfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } health_report { overall_health: UNKNOWN } } }
I20260131 03:03:07.542501 14937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.539947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.6.22.66:41925)
0131 03:03:07.540088 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.540092 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.540105 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:03:07.540225 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.540228 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.540283 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.540350 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:03:07.540721 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.541197 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.541840 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.541975 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.542017 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.542089 (+ 72us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.542151 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.542260 (+ 109us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.542305 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260131 03:03:07.544775 14580 raft_consensus.cc:1275] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Refusing update from remote peer e6de748b72ca40209a51efb372bf1cfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:07.544821 14864 raft_consensus.cc:1275] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Refusing update from remote peer e6de748b72ca40209a51efb372bf1cfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:07.545296 14946 consensus_queue.cc:1048] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:07.545424 14969 consensus_queue.cc:1048] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:07.553287 14418 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:03:07.553459 14418 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:03:07.555397 14418 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d: Sending DeleteTablet for 3 replicas of tablet 8a3fceae8dc04affa0b9b37111228cfb
I20260131 03:03:07.556226 14702 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a3fceae8dc04affa0b9b37111228cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:07 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:49936
I20260131 03:03:07.556321 14558 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a3fceae8dc04affa0b9b37111228cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:07 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:57384
I20260131 03:03:07.556450 14963 tablet_replica.cc:333] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb: stopping tablet replica
I20260131 03:03:07.556444 14844 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a3fceae8dc04affa0b9b37111228cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:07 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:38724
I20260131 03:03:07.556588 14963 raft_consensus.cc:2243] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:03:07.556630 14962 tablet_replica.cc:333] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6: stopping tablet replica
I20260131 03:03:07.556630 14961 tablet_replica.cc:333] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8: stopping tablet replica
I20260131 03:03:07.556793 14962 raft_consensus.cc:2243] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:03:07.556818 14961 raft_consensus.cc:2243] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:03:07.556888 14963 raft_consensus.cc:2272] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:07.556993 14962 raft_consensus.cc:2272] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:07.556979 14961 raft_consensus.cc:2272] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:07.557932 14962 ts_tablet_manager.cc:1916] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:07.557986 14963 ts_tablet_manager.cc:1916] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:07.560014 14963 ts_tablet_manager.cc:1929] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:03:07.560128 14963 log.cc:1199] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/8a3fceae8dc04affa0b9b37111228cfb
I20260131 03:03:07.560335 14962 ts_tablet_manager.cc:1929] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:03:07.560408 14962 log.cc:1199] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/8a3fceae8dc04affa0b9b37111228cfb
I20260131 03:03:07.560412 14963 ts_tablet_manager.cc:1950] T 8a3fceae8dc04affa0b9b37111228cfb P e6de748b72ca40209a51efb372bf1cfb: Deleting consensus metadata
I20260131 03:03:07.560796 14962 ts_tablet_manager.cc:1950] T 8a3fceae8dc04affa0b9b37111228cfb P 6110ec810fd74ff2adf98a640f84b2e6: Deleting consensus metadata
I20260131 03:03:07.561347 14407 catalog_manager.cc:4985] TS e6de748b72ca40209a51efb372bf1cfb (127.6.22.66:41925): tablet 8a3fceae8dc04affa0b9b37111228cfb (table test-table [id=a5fc64c0d1244c28acb5cc242736fcec]) successfully deleted
I20260131 03:03:07.561738 14407 catalog_manager.cc:4985] TS 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951): tablet 8a3fceae8dc04affa0b9b37111228cfb (table test-table [id=a5fc64c0d1244c28acb5cc242736fcec]) successfully deleted
I20260131 03:03:07.562220 14961 ts_tablet_manager.cc:1916] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:07.564123 14961 ts_tablet_manager.cc:1929] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:03:07.564193 14961 log.cc:1199] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-0/wal/wals/8a3fceae8dc04affa0b9b37111228cfb
I20260131 03:03:07.564442 14961 ts_tablet_manager.cc:1950] T 8a3fceae8dc04affa0b9b37111228cfb P f8e846a385ec4544ad7ecde4a1336cf8: Deleting consensus metadata
I20260131 03:03:07.565385 14407 catalog_manager.cc:4985] TS f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65:44813): tablet 8a3fceae8dc04affa0b9b37111228cfb (table test-table [id=a5fc64c0d1244c28acb5cc242736fcec]) successfully deleted
I20260131 03:03:07.565796 14418 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:03:07.566401 14418 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:03:07.573334 14558 tablet_service.cc:1505] Processing CreateTablet for tablet dff277c92b234e62b95f3ef36f6fe7ef (DEFAULT_TABLE table=test-table [id=97ed5d7555784dee97770f4d92d7403a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:07.573436 14702 tablet_service.cc:1505] Processing CreateTablet for tablet dff277c92b234e62b95f3ef36f6fe7ef (DEFAULT_TABLE table=test-table [id=97ed5d7555784dee97770f4d92d7403a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:07.573575 14844 tablet_service.cc:1505] Processing CreateTablet for tablet dff277c92b234e62b95f3ef36f6fe7ef (DEFAULT_TABLE table=test-table [id=97ed5d7555784dee97770f4d92d7403a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:07.573707 14558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff277c92b234e62b95f3ef36f6fe7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:07.573768 14702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff277c92b234e62b95f3ef36f6fe7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:07.573851 14844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff277c92b234e62b95f3ef36f6fe7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:07.575728 14938 tablet_bootstrap.cc:492] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb: Bootstrap starting.
I20260131 03:03:07.576234 14939 tablet_bootstrap.cc:492] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8: Bootstrap starting.
I20260131 03:03:07.576800 14938 tablet_bootstrap.cc:654] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:07.577096 14939 tablet_bootstrap.cc:654] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:07.577718 14940 tablet_bootstrap.cc:492] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6: Bootstrap starting.
I20260131 03:03:07.577936 14938 tablet_bootstrap.cc:492] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb: No bootstrap required, opened a new log
I20260131 03:03:07.578055 14938 ts_tablet_manager.cc:1403] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:03:07.578492 14938 raft_consensus.cc:359] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.578598 14940 tablet_bootstrap.cc:654] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:07.578635 14938 raft_consensus.cc:385] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:07.578711 14938 raft_consensus.cc:740] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6de748b72ca40209a51efb372bf1cfb, State: Initialized, Role: FOLLOWER
I20260131 03:03:07.578891 14938 consensus_queue.cc:260] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.579208 14938 ts_tablet_manager.cc:1434] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260131 03:03:07.579548 14939 tablet_bootstrap.cc:492] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8: No bootstrap required, opened a new log
I20260131 03:03:07.579671 14939 ts_tablet_manager.cc:1403] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260131 03:03:07.580096 14939 raft_consensus.cc:359] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.580214 14939 raft_consensus.cc:385] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:07.580260 14939 raft_consensus.cc:740] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e846a385ec4544ad7ecde4a1336cf8, State: Initialized, Role: FOLLOWER
I20260131 03:03:07.580411 14939 consensus_queue.cc:260] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.580741 14939 ts_tablet_manager.cc:1434] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260131 03:03:07.581928 14940 tablet_bootstrap.cc:492] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6: No bootstrap required, opened a new log
I20260131 03:03:07.582015 14940 ts_tablet_manager.cc:1403] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260131 03:03:07.582526 14940 raft_consensus.cc:359] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.582651 14940 raft_consensus.cc:385] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:07.582687 14940 raft_consensus.cc:740] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6110ec810fd74ff2adf98a640f84b2e6, State: Initialized, Role: FOLLOWER
I20260131 03:03:07.582825 14940 consensus_queue.cc:260] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.583174 14940 ts_tablet_manager.cc:1434] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260131 03:03:07.887305 14949 raft_consensus.cc:493] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:03:07.887506 14949 raft_consensus.cc:515] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.887984 14949 leader_election.cc:290] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6de748b72ca40209a51efb372bf1cfb (127.6.22.66:41925), 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951)
I20260131 03:03:07.888648 14721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dff277c92b234e62b95f3ef36f6fe7ef" candidate_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6de748b72ca40209a51efb372bf1cfb" is_pre_election: true
I20260131 03:03:07.888795 14864 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dff277c92b234e62b95f3ef36f6fe7ef" candidate_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6110ec810fd74ff2adf98a640f84b2e6" is_pre_election: true
I20260131 03:03:07.888846 14721 raft_consensus.cc:2468] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e846a385ec4544ad7ecde4a1336cf8 in term 0.
I20260131 03:03:07.888967 14864 raft_consensus.cc:2468] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e846a385ec4544ad7ecde4a1336cf8 in term 0.
I20260131 03:03:07.889252 14513 leader_election.cc:304] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6de748b72ca40209a51efb372bf1cfb, f8e846a385ec4544ad7ecde4a1336cf8; no voters:
I20260131 03:03:07.889442 14949 raft_consensus.cc:2804] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:03:07.889503 14949 raft_consensus.cc:493] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:03:07.889549 14949 raft_consensus.cc:3060] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:07.890544 14949 raft_consensus.cc:515] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.890929 14949 leader_election.cc:290] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [CANDIDATE]: Term 1 election: Requested vote from peers e6de748b72ca40209a51efb372bf1cfb (127.6.22.66:41925), 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951)
I20260131 03:03:07.891392 14721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dff277c92b234e62b95f3ef36f6fe7ef" candidate_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6de748b72ca40209a51efb372bf1cfb"
I20260131 03:03:07.891392 14864 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dff277c92b234e62b95f3ef36f6fe7ef" candidate_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6110ec810fd74ff2adf98a640f84b2e6"
I20260131 03:03:07.891523 14864 raft_consensus.cc:3060] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:07.891523 14721 raft_consensus.cc:3060] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:07.892612 14864 raft_consensus.cc:2468] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e846a385ec4544ad7ecde4a1336cf8 in term 1.
I20260131 03:03:07.892751 14721 raft_consensus.cc:2468] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e846a385ec4544ad7ecde4a1336cf8 in term 1.
I20260131 03:03:07.893116 14513 leader_election.cc:304] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6110ec810fd74ff2adf98a640f84b2e6, f8e846a385ec4544ad7ecde4a1336cf8; no voters:
I20260131 03:03:07.893311 14949 raft_consensus.cc:2804] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:03:07.893487 14949 raft_consensus.cc:697] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 LEADER]: Becoming Leader. State: Replica: f8e846a385ec4544ad7ecde4a1336cf8, State: Running, Role: LEADER
I20260131 03:03:07.893669 14949 consensus_queue.cc:237] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } }
I20260131 03:03:07.895547 14420 catalog_manager.cc:5654] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65). New cstate: current_term: 1 leader_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e846a385ec4544ad7ecde4a1336cf8" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 44813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 } health_report { overall_health: UNKNOWN } } }
I20260131 03:03:07.960776 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.958220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.6.22.65:44813)
0131 03:03:07.958458 (+ 238us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.958462 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.958474 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:03:07.958517 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.958519 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.958566 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.958642 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:03:07.958964 (+ 322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.959462 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.960089 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.960251 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.960295 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.960369 (+ 74us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.960432 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.960544 (+ 112us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.960589 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":95,"threads_started":1}
I20260131 03:03:07.972311 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.969435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:33951 (local address 127.6.22.65:53033)
0131 03:03:07.969768 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:07.969784 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:07.969895 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:07.970194 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:07.970196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:07.970203 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:03:07.970414 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.970419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.971199 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.971201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:07.971610 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.971616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.971710 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.971754 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.972063 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.972067 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:03:07.972119 (+ 52us) client_negotiation.cc:240] Negotiation successful
0131 03:03:07.972187 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":103,"threads_started":1}
I20260131 03:03:07.972318 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.969547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:53033 (local address 127.6.22.67:33951)
0131 03:03:07.969816 (+ 269us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.969820 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.969839 (+ 19us) server_negotiation.cc:408] Connection header received
0131 03:03:07.970023 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.970026 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.970084 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.970170 (+ 86us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:03:07.970591 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.971097 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.971718 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.971863 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.971901 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.971978 (+ 77us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.972048 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.972154 (+ 106us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.972204 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
I20260131 03:03:07.978442 14721 raft_consensus.cc:1275] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Refusing update from remote peer f8e846a385ec4544ad7ecde4a1336cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:07.978482 14864 raft_consensus.cc:1275] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Refusing update from remote peer f8e846a385ec4544ad7ecde4a1336cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:07.978922 15021 consensus_queue.cc:1048] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6110ec810fd74ff2adf98a640f84b2e6" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:07.979053 14949 consensus_queue.cc:1048] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6de748b72ca40209a51efb372bf1cfb" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:07.997423 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.994291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:56037 (local address 127.6.22.65:44813)
0131 03:03:07.994415 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:07.994420 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:07.994676 (+ 256us) server_negotiation.cc:408] Connection header received
0131 03:03:07.994860 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:07.994863 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:07.994910 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:07.994993 (+ 83us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:03:07.995427 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.995912 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.996574 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.996845 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.996900 (+ 55us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.996998 (+ 98us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.997101 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:07.997223 (+ 122us) server_negotiation.cc:300] Negotiation successful
0131 03:03:07.997276 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260131 03:03:07.997426 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:07.994152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:44813 (local address 127.6.22.67:56037)
0131 03:03:07.994580 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:07.994590 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:07.994724 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:07.995019 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:07.995022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:07.995031 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:03:07.995304 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.995312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.996029 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:07.996036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:07.996452 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:07.996459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:07.996592 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:07.996627 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:03:07.997108 (+ 481us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:03:07.997116 (+ 8us) client_negotiation.cc:769] Sending connection context
0131 03:03:07.997209 (+ 93us) client_negotiation.cc:240] Negotiation successful
0131 03:03:07.997292 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":128,"threads_started":1}
I20260131 03:03:08.020236 14420 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:03:08.020385 14420 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:03:08.022565 14420 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 650af9a4f6f14165bb9971ba0813c28d: Sending DeleteTablet for 3 replicas of tablet dff277c92b234e62b95f3ef36f6fe7ef
I20260131 03:03:08.023257 14558 tablet_service.cc:1552] Processing DeleteTablet for tablet dff277c92b234e62b95f3ef36f6fe7ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:08 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:57384
I20260131 03:03:08.023391 14702 tablet_service.cc:1552] Processing DeleteTablet for tablet dff277c92b234e62b95f3ef36f6fe7ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:08 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:49936
I20260131 03:03:08.023526 14844 tablet_service.cc:1552] Processing DeleteTablet for tablet dff277c92b234e62b95f3ef36f6fe7ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:08 UTC) from {username='oryx', principal='oryx/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:38724
I20260131 03:03:08.023563 14961 tablet_replica.cc:333] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8: stopping tablet replica
I20260131 03:03:08.023597 14963 tablet_replica.cc:333] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb: stopping tablet replica
I20260131 03:03:08.023695 14961 raft_consensus.cc:2243] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:03:08.023705 14963 raft_consensus.cc:2243] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:03:08.023726 14962 tablet_replica.cc:333] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6: stopping tablet replica
I20260131 03:03:08.023833 14963 raft_consensus.cc:2272] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:08.023861 14962 raft_consensus.cc:2243] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:03:08.023910 14961 raft_consensus.cc:2272] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:08.024017 14962 raft_consensus.cc:2272] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:08.024698 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 14495
I20260131 03:03:08.024717 14963 ts_tablet_manager.cc:1916] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:08.024876 14961 ts_tablet_manager.cc:1916] T dff277c92b234e62b95f3ef36f6fe7ef P f8e846a385ec4544ad7ecde4a1336cf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:08.024876 14962 ts_tablet_manager.cc:1916] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:08.027211 14963 ts_tablet_manager.cc:1929] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:03:08.027285 14963 log.cc:1199] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/dff277c92b234e62b95f3ef36f6fe7ef
I20260131 03:03:08.027472 14963 ts_tablet_manager.cc:1950] T dff277c92b234e62b95f3ef36f6fe7ef P e6de748b72ca40209a51efb372bf1cfb: Deleting consensus metadata
I20260131 03:03:08.027612 14962 ts_tablet_manager.cc:1929] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:03:08.027726 14962 log.cc:1199] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/dff277c92b234e62b95f3ef36f6fe7ef
I20260131 03:03:08.028039 14962 ts_tablet_manager.cc:1950] T dff277c92b234e62b95f3ef36f6fe7ef P 6110ec810fd74ff2adf98a640f84b2e6: Deleting consensus metadata
I20260131 03:03:08.028276 14407 catalog_manager.cc:4985] TS e6de748b72ca40209a51efb372bf1cfb (127.6.22.66:41925): tablet dff277c92b234e62b95f3ef36f6fe7ef (table test-table [id=97ed5d7555784dee97770f4d92d7403a]) successfully deleted
I20260131 03:03:08.028800 14407 catalog_manager.cc:4985] TS 6110ec810fd74ff2adf98a640f84b2e6 (127.6.22.67:33951): tablet dff277c92b234e62b95f3ef36f6fe7ef (table test-table [id=97ed5d7555784dee97770f4d92d7403a]) successfully deleted
W20260131 03:03:08.035418 14407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.22.65:44813 (error 108)
I20260131 03:03:08.035683 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 14637
I20260131 03:03:08.037642 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:08.036639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:44813 (local address 127.0.0.1:57412)
0131 03:03:08.037375 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:08.037462 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.65:44813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":98,"threads_started":1}
W20260131 03:03:08.037794 14407 catalog_manager.cc:4712] TS f8e846a385ec4544ad7ecde4a1336cf8 (127.6.22.65:44813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dff277c92b234e62b95f3ef36f6fe7ef: Network error: Client connection negotiation failed: client connection to 127.6.22.65:44813: connect: Connection refused (error 111)
I20260131 03:03:08.044256 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 14779
I20260131 03:03:08.052232 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 14389
2026-01-31T03:03:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4627 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:03:08 dist-test-slave-j368 krb5kdc[15040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:03:08 dist-test-slave-j368 krb5kdc[15040](info): set up 2 sockets
Jan 31 03:03:08 dist-test-slave-j368 krb5kdc[15040](info): commencing operation
krb5kdc: starting...
W20260131 03:03:10.055322 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:03:10 dist-test-slave-j368 krb5kdc[15040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:03:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:03:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:03:10.206286 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:33275
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:38293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:33275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:10.321038 15056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:10.321296 15056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:10.321415 15056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:10.325054 15056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:03:10.325135 15056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:10.325160 15056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:10.325244 15056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:03:10.325279 15056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:03:10.330171 15056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:38293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:33275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:33275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:10.331439 15056 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:10.332367 15056 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:10.338568 15064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:10.338586 15061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:10.338569 15062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:10.339030 15056 server_base.cc:1060] running on GCE node
I20260131 03:03:10.339684 15056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:10.340776 15056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:10.341988 15056 hybrid_clock.cc:648] HybridClock initialized: now 1769828590341961 us; error 50 us; skew 500 ppm
W20260131 03:03:10.345654 15056 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-01-31T03:03:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2301 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:03:10 dist-test-slave-j368 krb5kdc[15069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:03:10 dist-test-slave-j368 krb5kdc[15069](info): set up 2 sockets
Jan 31 03:03:10 dist-test-slave-j368 krb5kdc[15069](info): commencing operation
krb5kdc: starting...
W20260131 03:03:12.386965 6233 mini_kdc.cc:121] Time spent starting KDC: real 2.008s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:03:12 dist-test-slave-j368 krb5kdc[15069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:03:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:03:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:03:12.535403 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:44187
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:44187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:12.646903 15085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:12.647236 15085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:12.647341 15085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:12.650894 15085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:03:12.651036 15085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:12.651082 15085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:12.651124 15085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:03:12.651170 15085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:03:12.655853 15085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:44187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:44187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:12.657160 15085 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:12.658107 15085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:12.664300 15090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:12.664302 15091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:12.664302 15093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:12.664708 15085 server_base.cc:1060] running on GCE node
I20260131 03:03:12.665264 15085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:12.666206 15085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:12.667395 15085 hybrid_clock.cc:648] HybridClock initialized: now 1769828592667353 us; error 66 us; skew 500 ppm
Jan 31 03:03:12 dist-test-slave-j368 krb5kdc[15069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828592, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:12.671211 15085 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:03:12.672610 15085 webserver.cc:492] Webserver started at http://127.6.22.126:45153/ using document root <none> and password file <none>
I20260131 03:03:12.673292 15085 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:12.673357 15085 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:12.673537 15085 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:12.675416 15085 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "5e16fcabcf6b4104aeb28fd1e6224f42"
format_stamp: "Formatted at 2026-01-31 03:03:12 on dist-test-slave-j368"
server_key: "b68fee1f6fad83b406a8bb8bc4eac5da"
server_key_iv: "af3206d9a81049e174e400daae73a5a2"
server_key_version: "encryptionkey@0"
I20260131 03:03:12.675928 15085 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "5e16fcabcf6b4104aeb28fd1e6224f42"
format_stamp: "Formatted at 2026-01-31 03:03:12 on dist-test-slave-j368"
server_key: "b68fee1f6fad83b406a8bb8bc4eac5da"
server_key_iv: "af3206d9a81049e174e400daae73a5a2"
server_key_version: "encryptionkey@0"
I20260131 03:03:12.680229 15085 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260131 03:03:12.682937 15100 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:12.684130 15085 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260131 03:03:12.684283 15085 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "5e16fcabcf6b4104aeb28fd1e6224f42"
format_stamp: "Formatted at 2026-01-31 03:03:12 on dist-test-slave-j368"
server_key: "b68fee1f6fad83b406a8bb8bc4eac5da"
server_key_iv: "af3206d9a81049e174e400daae73a5a2"
server_key_version: "encryptionkey@0"
I20260131 03:03:12.684394 15085 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:12.696875 15085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:12.699929 15085 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:12.700121 15085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:12.709510 15152 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:44187 every 8 connection(s)
I20260131 03:03:12.709510 15085 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:44187
I20260131 03:03:12.710870 15085 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:03:12.712838 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 15085
I20260131 03:03:12.712998 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:03:12.713317 6233 external_mini_cluster.cc:1468] Setting key 9ca5c4354587a99e2c8291a1eec0eff0
I20260131 03:03:12.714294 15153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:12.721827 15153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42: Bootstrap starting.
Jan 31 03:03:12 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:03:12.724702 15153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:12.725725 15153 log.cc:826] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42: Log is configured to *not* fsync() on all Append() calls
I20260131 03:03:12.728214 15153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42: No bootstrap required, opened a new log
I20260131 03:03:12.729391 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:12.715041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.6.22.126:44187)
0131 03:03:12.715613 (+ 572us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:12.715625 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:12.715671 (+ 46us) server_negotiation.cc:408] Connection header received
0131 03:03:12.717145 (+ 1474us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:12.717175 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:12.717603 (+ 428us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:12.718066 (+ 463us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:12.719048 (+ 982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:12.720220 (+ 1172us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:12.721051 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:12.721333 (+ 282us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:12.724076 (+ 2743us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:12.724091 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:12.724106 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:12.724166 (+ 60us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:12.726435 (+ 2269us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:12.727017 (+ 582us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:12.727023 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:12.727030 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:12.727128 (+ 98us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:12.727382 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:12.727386 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:12.727388 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:12.727847 (+ 459us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:12.728092 (+ 245us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:12.728279 (+ 187us) server_negotiation.cc:300] Negotiation successful
0131 03:03:12.728562 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":161,"threads_started":1}
I20260131 03:03:12.731549 15153 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 44187 } }
I20260131 03:03:12.731807 15153 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:12.731878 15153 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e16fcabcf6b4104aeb28fd1e6224f42, State: Initialized, Role: FOLLOWER
I20260131 03:03:12.732354 15153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 44187 } }
I20260131 03:03:12.732510 15153 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:03:12.732574 15153 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:03:12.732722 15153 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:12.733678 15153 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 44187 } }
I20260131 03:03:12.734079 15153 leader_election.cc:304] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e16fcabcf6b4104aeb28fd1e6224f42; no voters:
I20260131 03:03:12.734426 15153 leader_election.cc:290] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:03:12.734723 15158 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:03:12.734952 15158 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 1 LEADER]: Becoming Leader. State: Replica: 5e16fcabcf6b4104aeb28fd1e6224f42, State: Running, Role: LEADER
I20260131 03:03:12.735287 15158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 44187 } }
I20260131 03:03:12.735677 15153 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:03:12.737037 15159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 44187 } } }
I20260131 03:03:12.737088 15160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e16fcabcf6b4104aeb28fd1e6224f42. Latest consensus state: current_term: 1 leader_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 44187 } } }
I20260131 03:03:12.737206 15160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:12.737206 15159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [sys.catalog]: This master's current role is: LEADER
W20260131 03:03:12.741101 15173 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:03:12.741285 15173 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:03:12.741364 15167 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:03:12.742130 15167 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:03:12.748853 15167 catalog_manager.cc:1357] Generated new cluster ID: 90a5929442f444be89345b6a918f53d6
I20260131 03:03:12.748994 15167 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:03:12.769461 15167 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:03:12.770581 15167 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:03:12.778179 15167 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42: Generated new TSK 0
I20260131 03:03:12.779047 15167 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:03:12.835908 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:44187
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:12.972457 15181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:12.972759 15181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:12.972886 15181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:12.976404 15181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:12.976485 15181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:12.976576 15181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:03:12.981264 15181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:12.982661 15181 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:12.983732 15181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:12.991796 15186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:12.991828 15189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:12.991796 15187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:12.992269 15181 server_base.cc:1060] running on GCE node
I20260131 03:03:12.992851 15181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:12.993491 15181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:12.994668 15181 hybrid_clock.cc:648] HybridClock initialized: now 1769828592994641 us; error 49 us; skew 500 ppm
Jan 31 03:03:12 dist-test-slave-j368 krb5kdc[15069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828592, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:12.997534 15181 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:03:12.998759 15181 webserver.cc:492] Webserver started at http://127.6.22.65:35079/ using document root <none> and password file <none>
I20260131 03:03:12.999382 15181 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:12.999435 15181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:12.999614 15181 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:13.001569 15181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "f1c46ceddbf94d1983924505579575f2"
format_stamp: "Formatted at 2026-01-31 03:03:12 on dist-test-slave-j368"
server_key: "941654870ab4e795d025b3e15c7c5aa7"
server_key_iv: "069bb8781076f3db9e3c7fda7c04af46"
server_key_version: "encryptionkey@0"
I20260131 03:03:13.002049 15181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "f1c46ceddbf94d1983924505579575f2"
format_stamp: "Formatted at 2026-01-31 03:03:12 on dist-test-slave-j368"
server_key: "941654870ab4e795d025b3e15c7c5aa7"
server_key_iv: "069bb8781076f3db9e3c7fda7c04af46"
server_key_version: "encryptionkey@0"
I20260131 03:03:13.005906 15181 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260131 03:03:13.008860 15196 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:13.010437 15181 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:03:13.010586 15181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "f1c46ceddbf94d1983924505579575f2"
format_stamp: "Formatted at 2026-01-31 03:03:12 on dist-test-slave-j368"
server_key: "941654870ab4e795d025b3e15c7c5aa7"
server_key_iv: "069bb8781076f3db9e3c7fda7c04af46"
server_key_version: "encryptionkey@0"
I20260131 03:03:13.010690 15181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:13.029489 15181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:13.032760 15181 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:13.032945 15181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:13.033576 15181 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:13.034523 15181 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:13.034574 15181 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:13.034615 15181 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:13.034637 15181 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:13.044808 15181 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:45009
I20260131 03:03:13.044842 15309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:45009 every 8 connection(s)
I20260131 03:03:13.045913 15181 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:03:13.053804 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 15181
I20260131 03:03:13.053969 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:03:13.054246 6233 external_mini_cluster.cc:1468] Setting key be3c7ead209ecdbffa0f99cb7656708d
Jan 31 03:03:13 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828592, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:03:13.061338 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:13.047957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:50257 (local address 127.6.22.126:44187)
0131 03:03:13.048135 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:13.048139 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:13.048980 (+ 841us) server_negotiation.cc:408] Connection header received
0131 03:03:13.049911 (+ 931us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:13.049915 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:13.049986 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:13.050100 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:13.051912 (+ 1812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:13.052432 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:13.053165 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:13.053347 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:13.056171 (+ 2824us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:13.056189 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:13.056192 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:13.056220 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:13.058119 (+ 1899us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:13.059079 (+ 960us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:13.059086 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:13.059091 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:13.059172 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:13.059845 (+ 673us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:13.059851 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:13.059855 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:13.060124 (+ 269us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:13.060257 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:13.060887 (+ 630us) server_negotiation.cc:300] Negotiation successful
0131 03:03:13.061083 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260131 03:03:13.061893 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:13.048268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:50257)
0131 03:03:13.048819 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:13.048861 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:13.049690 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:13.050284 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:13.050293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:13.050720 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:13.051711 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:13.051727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:13.052574 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:13.052579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:13.053023 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:13.053031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:13.053329 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:13.054004 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:13.054067 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:13.055974 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:13.058415 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:13.058424 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:13.058441 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:13.058872 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:13.059512 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:13.059518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:13.059531 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:13.059713 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:13.060254 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:13.060260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:13.060545 (+ 285us) client_negotiation.cc:769] Sending connection context
0131 03:03:13.060805 (+ 260us) client_negotiation.cc:240] Negotiation successful
0131 03:03:13.061080 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":115,"threads_started":1}
I20260131 03:03:13.063299 15310 heartbeater.cc:344] Connected to a master server at 127.6.22.126:44187
I20260131 03:03:13.063611 15310 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
I20260131 03:03:13.064246 15310 heartbeater.cc:507] Master 127.6.22.126:44187 requested a full tablet report, sending...
I20260131 03:03:13.066465 15117 ts_manager.cc:194] Registered new tserver with Master: f1c46ceddbf94d1983924505579575f2 (127.6.22.65:45009)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
I20260131 03:03:13.068320 15117 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:50257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:03:13.109978 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:44187
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:13.228103 15317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:13.228410 15317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:13.228502 15317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:13.232095 15317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:13.232234 15317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:13.232362 15317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:03:13.237346 15317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:13.238780 15317 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:13.239768 15317 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:13.247303 15323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:13.247310 15325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:13.247306 15322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:13.247466 15317 server_base.cc:1060] running on GCE node
I20260131 03:03:13.247994 15317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:13.248622 15317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:13.249861 15317 hybrid_clock.cc:648] HybridClock initialized: now 1769828593249834 us; error 45 us; skew 500 ppm
Jan 31 03:03:13 dist-test-slave-j368 krb5kdc[15069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828593, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:13.252861 15317 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:03:13.254074 15317 webserver.cc:492] Webserver started at http://127.6.22.66:38333/ using document root <none> and password file <none>
I20260131 03:03:13.254706 15317 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:13.254788 15317 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:13.255007 15317 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:13.257002 15317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "fff67c69125142ff8cceb1f1e238843d"
format_stamp: "Formatted at 2026-01-31 03:03:13 on dist-test-slave-j368"
server_key: "7dcc3242f5dcd0b85c5e3d92c9402e70"
server_key_iv: "bfa5a0a07983ea07551c60cf388aefdc"
server_key_version: "encryptionkey@0"
I20260131 03:03:13.257529 15317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "fff67c69125142ff8cceb1f1e238843d"
format_stamp: "Formatted at 2026-01-31 03:03:13 on dist-test-slave-j368"
server_key: "7dcc3242f5dcd0b85c5e3d92c9402e70"
server_key_iv: "bfa5a0a07983ea07551c60cf388aefdc"
server_key_version: "encryptionkey@0"
I20260131 03:03:13.261400 15317 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260131 03:03:13.263999 15332 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:13.265188 15317 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260131 03:03:13.265360 15317 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "fff67c69125142ff8cceb1f1e238843d"
format_stamp: "Formatted at 2026-01-31 03:03:13 on dist-test-slave-j368"
server_key: "7dcc3242f5dcd0b85c5e3d92c9402e70"
server_key_iv: "bfa5a0a07983ea07551c60cf388aefdc"
server_key_version: "encryptionkey@0"
I20260131 03:03:13.265483 15317 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:13.278702 15317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:13.282054 15317 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:13.282285 15317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:13.282869 15317 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:13.283815 15317 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:13.283893 15317 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:13.283965 15317 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:13.284018 15317 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:13.293968 15317 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:39595
I20260131 03:03:13.293983 15445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:39595 every 8 connection(s)
I20260131 03:03:13.295039 15317 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:03:13.297004 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 15317
I20260131 03:03:13.297132 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:03:13.297430 6233 external_mini_cluster.cc:1468] Setting key 57e61868dff6fa92767417b8e36a045a
Jan 31 03:03:13 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828593, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
I20260131 03:03:13.309607 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:13.297030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:58421 (local address 127.6.22.126:44187)
0131 03:03:13.297218 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:13.297223 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:13.298066 (+ 843us) server_negotiation.cc:408] Connection header received
0131 03:03:13.299013 (+ 947us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:13.299017 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:13.299083 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:13.299161 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:13.301005 (+ 1844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:13.301539 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:13.302244 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:13.302440 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:13.305229 (+ 2789us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:13.305252 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:13.305254 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:13.305288 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:13.306801 (+ 1513us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:13.307583 (+ 782us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:13.307587 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:13.307589 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:13.307648 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:13.308139 (+ 491us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:13.308143 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:13.308145 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:13.308426 (+ 281us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:13.308542 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:13.309331 (+ 789us) server_negotiation.cc:300] Negotiation successful
0131 03:03:13.309471 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260131 03:03:13.310667 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:13.297388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:58421)
0131 03:03:13.297906 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:13.297954 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:13.298802 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:13.299335 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:13.299344 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:13.299752 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:13.300823 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:13.300841 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:13.301684 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:13.301689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:13.302108 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:13.302116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:13.302429 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:13.303118 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:13.303147 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:13.305018 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:13.307154 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:13.307160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:13.307172 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:13.307457 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:13.307819 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:13.307829 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:13.307831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:13.308017 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:13.308575 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:13.308587 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:13.308988 (+ 401us) client_negotiation.cc:769] Sending connection context
0131 03:03:13.309286 (+ 298us) client_negotiation.cc:240] Negotiation successful
0131 03:03:13.309644 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":165,"threads_started":1}
I20260131 03:03:13.311923 15446 heartbeater.cc:344] Connected to a master server at 127.6.22.126:44187
I20260131 03:03:13.312214 15446 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:13.312896 15446 heartbeater.cc:507] Master 127.6.22.126:44187 requested a full tablet report, sending...
I20260131 03:03:13.314137 15117 ts_manager.cc:194] Registered new tserver with Master: fff67c69125142ff8cceb1f1e238843d (127.6.22.66:39595)
I20260131 03:03:13.314993 15117 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:58421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:03:13.347540 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:44187
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:13.461062 15453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:13.461334 15453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:13.461452 15453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:13.465056 15453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:13.465131 15453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:13.465286 15453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:03:13.470186 15453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:13.471436 15453 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:13.472378 15453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:13.479501 15458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:13.479449 15459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:13.479657 15453 server_base.cc:1060] running on GCE node
W20260131 03:03:13.479445 15461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:13.480217 15453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:13.480926 15453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:13.482127 15453 hybrid_clock.cc:648] HybridClock initialized: now 1769828593482114 us; error 33 us; skew 500 ppm
Jan 31 03:03:13 dist-test-slave-j368 krb5kdc[15069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828593, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:13.485562 15453 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:03:13.486837 15453 webserver.cc:492] Webserver started at http://127.6.22.67:42743/ using document root <none> and password file <none>
I20260131 03:03:13.487478 15453 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:13.487563 15453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:13.487799 15453 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:13.489832 15453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "1431943f8b194571a1f560019f5d8c53"
format_stamp: "Formatted at 2026-01-31 03:03:13 on dist-test-slave-j368"
server_key: "e0b6068ed716b242ac282917e4a97d12"
server_key_iv: "56860f4a1a35ead5f23bc2b9acbefd54"
server_key_version: "encryptionkey@0"
I20260131 03:03:13.490396 15453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "1431943f8b194571a1f560019f5d8c53"
format_stamp: "Formatted at 2026-01-31 03:03:13 on dist-test-slave-j368"
server_key: "e0b6068ed716b242ac282917e4a97d12"
server_key_iv: "56860f4a1a35ead5f23bc2b9acbefd54"
server_key_version: "encryptionkey@0"
I20260131 03:03:13.494205 15453 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260131 03:03:13.496742 15468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:13.497917 15453 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260131 03:03:13.498080 15453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "1431943f8b194571a1f560019f5d8c53"
format_stamp: "Formatted at 2026-01-31 03:03:13 on dist-test-slave-j368"
server_key: "e0b6068ed716b242ac282917e4a97d12"
server_key_iv: "56860f4a1a35ead5f23bc2b9acbefd54"
server_key_version: "encryptionkey@0"
I20260131 03:03:13.498206 15453 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:13.512984 15453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:13.516297 15453 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:13.516517 15453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:13.517194 15453 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:13.518218 15453 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:13.518301 15453 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:13.518365 15453 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:13.518415 15453 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:13.528434 15453 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:36913
I20260131 03:03:13.528460 15581 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:36913 every 8 connection(s)
I20260131 03:03:13.529595 15453 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:03:13.534543 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 15453
I20260131 03:03:13.534724 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:03:13.534976 6233 external_mini_cluster.cc:1468] Setting key ca9c2ca4fd3c98688602033dce835738
Jan 31 03:03:13 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828593, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:03:13.544072 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:13.531664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:47807 (local address 127.6.22.126:44187)
0131 03:03:13.531831 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:13.531835 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:13.532590 (+ 755us) server_negotiation.cc:408] Connection header received
0131 03:03:13.533505 (+ 915us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:13.533509 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:13.533565 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:13.533648 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:13.535614 (+ 1966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:13.536352 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:13.537205 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:13.537402 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:13.539914 (+ 2512us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:13.539937 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:13.539940 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:13.539972 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:13.541583 (+ 1611us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:13.542130 (+ 547us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:13.542133 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:13.542134 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:13.542198 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:13.542793 (+ 595us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:13.542796 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:13.542797 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:13.542985 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:13.543107 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:13.543796 (+ 689us) server_negotiation.cc:300] Negotiation successful
0131 03:03:13.543926 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260131 03:03:13.544888 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:13.531949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:47807)
0131 03:03:13.532417 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:13.532457 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:13.533301 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:13.533818 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:13.533826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:13.534268 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:13.535402 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:13.535422 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:13.536488 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:13.536494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:13.537080 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:13.537091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:13.537369 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:13.538010 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:13.538040 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:13.539725 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:13.541724 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:13.541729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:13.541741 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:13.542008 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:13.542407 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:13.542413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:13.542415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:13.542634 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:13.543191 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:13.543198 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:13.543500 (+ 302us) client_negotiation.cc:769] Sending connection context
0131 03:03:13.543748 (+ 248us) client_negotiation.cc:240] Negotiation successful
0131 03:03:13.544011 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":118,"threads_started":1}
I20260131 03:03:13.546205 15582 heartbeater.cc:344] Connected to a master server at 127.6.22.126:44187
I20260131 03:03:13.546506 15582 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:13.547192 15582 heartbeater.cc:507] Master 127.6.22.126:44187 requested a full tablet report, sending...
I20260131 03:03:13.548492 15116 ts_manager.cc:194] Registered new tserver with Master: 1431943f8b194571a1f560019f5d8c53 (127.6.22.67:36913)
I20260131 03:03:13.549343 15116 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:47807
I20260131 03:03:13.549487 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/oryx_127.6.22.65.keytab.
Entry for principal oryx/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/oryx_127.6.22.65.keytab.
WARNING: no policy specified for oryx/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/oryx_127.6.22.66.keytab.
Entry for principal oryx/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/oryx_127.6.22.66.keytab.
WARNING: no policy specified for oryx/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/oryx_127.6.22.67.keytab.
Entry for principal oryx/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/oryx_127.6.22.67.keytab.
I20260131 03:03:13.624706 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 15181
I20260131 03:03:13.632189 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 15317
I20260131 03:03:13.639037 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 15453
I20260131 03:03:13.645085 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 15085
I20260131 03:03:13.652367 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:44187
--webserver_interface=127.6.22.126
--webserver_port=45153
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:44187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:13.767040 15591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:13.767314 15591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:13.767374 15591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:13.770983 15591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:03:13.771092 15591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:13.771117 15591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:13.771137 15591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:03:13.771157 15591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:03:13.776084 15591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:44187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:44187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=45153
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:13.777388 15591 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:13.778345 15591 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:13.784437 15597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:13.784423 15596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:13.784423 15599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:13.784886 15591 server_base.cc:1060] running on GCE node
I20260131 03:03:13.785440 15591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:13.786484 15591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:13.787680 15591 hybrid_clock.cc:648] HybridClock initialized: now 1769828593787657 us; error 50 us; skew 500 ppm
Jan 31 03:03:13 dist-test-slave-j368 krb5kdc[15069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828593, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:13.790513 15591 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:03:13.791762 15591 webserver.cc:492] Webserver started at http://127.6.22.126:45153/ using document root <none> and password file <none>
I20260131 03:03:13.792374 15591 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:13.792470 15591 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:13.796381 15591 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:03:13.798684 15606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:13.799794 15591 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:03:13.799969 15591 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "5e16fcabcf6b4104aeb28fd1e6224f42"
format_stamp: "Formatted at 2026-01-31 03:03:12 on dist-test-slave-j368"
server_key: "b68fee1f6fad83b406a8bb8bc4eac5da"
server_key_iv: "af3206d9a81049e174e400daae73a5a2"
server_key_version: "encryptionkey@0"
I20260131 03:03:13.800472 15591 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:13.819797 15591 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:13.822806 15591 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:13.823031 15591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:13.830936 15591 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:44187
I20260131 03:03:13.830936 15658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:44187 every 8 connection(s)
I20260131 03:03:13.832186 15591 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:03:13.837235 15659 sys_catalog.cc:263] Verifying existing consensus state
I20260131 03:03:13.839099 15659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42: Bootstrap starting.
I20260131 03:03:13.839243 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 15591
I20260131 03:03:13.839709 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:45009
--local_ip_for_outbound_sockets=127.6.22.65
--tserver_master_addrs=127.6.22.126:44187
--webserver_port=35079
--webserver_interface=127.6.22.65
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.22.65
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/oryx_127.6.22.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:03:13.861560 15659 log.cc:826] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42: Log is configured to *not* fsync() on all Append() calls
I20260131 03:03:13.870441 15659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:03:13.870915 15659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42: Bootstrap complete.
I20260131 03:03:13.875674 15659 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 44187 } }
I20260131 03:03:13.876190 15659 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e16fcabcf6b4104aeb28fd1e6224f42, State: Initialized, Role: FOLLOWER
I20260131 03:03:13.876819 15659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 44187 } }
I20260131 03:03:13.876976 15659 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:03:13.877050 15659 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:03:13.877151 15659 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 1 FOLLOWER]: Advancing to term 2
I20260131 03:03:13.879415 15659 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 44187 } }
I20260131 03:03:13.879848 15659 leader_election.cc:304] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e16fcabcf6b4104aeb28fd1e6224f42; no voters:
I20260131 03:03:13.880228 15659 leader_election.cc:290] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260131 03:03:13.880412 15663 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 2 FOLLOWER]: Leader election won for term 2
I20260131 03:03:13.880771 15663 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [term 2 LEADER]: Becoming Leader. State: Replica: 5e16fcabcf6b4104aeb28fd1e6224f42, State: Running, Role: LEADER
I20260131 03:03:13.881206 15663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 44187 } }
I20260131 03:03:13.881680 15659 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:03:13.882934 15665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e16fcabcf6b4104aeb28fd1e6224f42. Latest consensus state: current_term: 2 leader_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 44187 } } }
I20260131 03:03:13.883095 15665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:13.883371 15664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e16fcabcf6b4104aeb28fd1e6224f42" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 44187 } } }
I20260131 03:03:13.883456 15664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42 [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:13.883482 15672 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:03:13.887176 15672 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:03:13.888279 15672 catalog_manager.cc:1269] Loaded cluster ID: 90a5929442f444be89345b6a918f53d6
I20260131 03:03:13.888377 15672 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:03:13.890358 15672 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:03:13.891106 15672 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 5e16fcabcf6b4104aeb28fd1e6224f42: Loaded TSK: 0
I20260131 03:03:13.892223 15672 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260131 03:03:13.994599 15661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:13.994882 15661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:13.994946 15661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:13.998517 15661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:13.998594 15661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:13.998721 15661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:03:14.003293 15661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/oryx_127.6.22.65.keytab
--principal=oryx/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:45009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=35079
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:14.004719 15661 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:14.005929 15661 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:14.015432 15688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:14.015424 15686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:14.016350 15685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:14.017333 15661 server_base.cc:1060] running on GCE node
I20260131 03:03:14.017915 15661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:14.018733 15661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:14.019953 15661 hybrid_clock.cc:648] HybridClock initialized: now 1769828594019931 us; error 41 us; skew 500 ppm
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828594, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:14.023418 15661 init.cc:377] Logged in from keytab as oryx/127.6.22.65@KRBTEST.COM (short username oryx)
I20260131 03:03:14.024953 15661 webserver.cc:492] Webserver started at http://127.6.22.65:35079/ using document root <none> and password file <none>
I20260131 03:03:14.025651 15661 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:14.025750 15661 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:14.030166 15661 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260131 03:03:14.032469 15695 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:14.033793 15661 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260131 03:03:14.033905 15661 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "f1c46ceddbf94d1983924505579575f2"
format_stamp: "Formatted at 2026-01-31 03:03:12 on dist-test-slave-j368"
server_key: "941654870ab4e795d025b3e15c7c5aa7"
server_key_iv: "069bb8781076f3db9e3c7fda7c04af46"
server_key_version: "encryptionkey@0"
I20260131 03:03:14.034431 15661 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:14.046234 15661 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:14.050081 15661 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:14.050343 15661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:14.051054 15661 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:14.052153 15661 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:14.052205 15661 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:14.052250 15661 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:14.052296 15661 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:14.064658 15661 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:45009
I20260131 03:03:14.064692 15808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:45009 every 8 connection(s)
I20260131 03:03:14.065836 15661 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:03:14.067862 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 15661
I20260131 03:03:14.068397 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:39595
--local_ip_for_outbound_sockets=127.6.22.66
--tserver_master_addrs=127.6.22.126:44187
--webserver_port=38333
--webserver_interface=127.6.22.66
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.22.66
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/oryx_127.6.22.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:14.081638 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:14.068172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:57243)
0131 03:03:14.069185 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:14.069231 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:14.070246 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:14.071438 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:14.071448 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:14.071976 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:14.073306 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.073329 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.075351 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.075355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:14.076102 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.076114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.076398 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.077312 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:14.077353 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:14.080619 (+ 3266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":146,"threads_started":1}
W20260131 03:03:14.081743 15811 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:14.068029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:57243 (local address 127.6.22.126:44187)
0131 03:03:14.068768 (+ 739us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:14.068780 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:14.069405 (+ 625us) server_negotiation.cc:408] Connection header received
0131 03:03:14.070714 (+ 1309us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:14.070726 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:14.071011 (+ 285us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:14.071287 (+ 276us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:14.073567 (+ 2280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.075138 (+ 1571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.076271 (+ 1133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.076650 (+ 379us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.080958 (+ 4308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:57243: BlockingRecv error: recv got EOF from 127.6.22.65:57243 (error 108)
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":137,"threads_started":1}
W20260131 03:03:14.082721 15809 heartbeater.cc:646] Failed to heartbeat to 127.6.22.126:44187 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.22.126:44187: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:14.089619 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:14.083386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:33539)
0131 03:03:14.083585 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:14.083601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:14.083695 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:14.084007 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:14.084010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:14.084214 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:14.084534 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.084547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.085711 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.085716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:14.086328 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.086341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.086473 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.087094 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:14.087119 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:14.089467 (+ 2348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20260131 03:03:14.089819 15811 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:14.083475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:33539 (local address 127.6.22.126:44187)
0131 03:03:14.083624 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:14.083630 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:14.083651 (+ 21us) server_negotiation.cc:408] Connection header received
0131 03:03:14.083805 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:14.083809 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:14.083868 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:14.083966 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:14.084738 (+ 772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.085541 (+ 803us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.086474 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.086668 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.089691 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:33539: BlockingRecv error: recv got EOF from 127.6.22.65:33539 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:14.096537 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:14.090667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:39733)
0131 03:03:14.091007 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:14.091022 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:14.091127 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:14.091421 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:14.091424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:14.091612 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:14.091853 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.091860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.092878 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.092882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:14.093488 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.093499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.093621 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.094178 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:14.094195 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:14.096391 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245}
W20260131 03:03:14.096756 15811 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:14.090760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:39733 (local address 127.6.22.126:44187)
0131 03:03:14.090887 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:14.090892 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:14.091111 (+ 219us) server_negotiation.cc:408] Connection header received
0131 03:03:14.091229 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:14.091233 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:14.091292 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:14.091393 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:14.091983 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.092757 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.093621 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.094152 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.096585 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:39733: BlockingRecv error: recv got EOF from 127.6.22.65:39733 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260131 03:03:14.097177 15809 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260131 03:03:14.193735 15813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:14.194010 15813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:14.194072 15813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:14.197621 15813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:14.197708 15813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:14.197794 15813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:03:14.202523 15813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/oryx_127.6.22.66.keytab
--principal=oryx/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:39595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=38333
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:14.203748 15813 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:14.204686 15813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:14.212338 15818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:14.212338 15819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:14.212338 15821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:14.212404 15813 server_base.cc:1060] running on GCE node
I20260131 03:03:14.213063 15813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:14.213696 15813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:14.214890 15813 hybrid_clock.cc:648] HybridClock initialized: now 1769828594214869 us; error 50 us; skew 500 ppm
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828594, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:14.217726 15813 init.cc:377] Logged in from keytab as oryx/127.6.22.66@KRBTEST.COM (short username oryx)
I20260131 03:03:14.218990 15813 webserver.cc:492] Webserver started at http://127.6.22.66:38333/ using document root <none> and password file <none>
I20260131 03:03:14.219661 15813 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:14.219765 15813 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:14.223846 15813 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260131 03:03:14.226120 15828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:14.227293 15813 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:03:14.227459 15813 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "fff67c69125142ff8cceb1f1e238843d"
format_stamp: "Formatted at 2026-01-31 03:03:13 on dist-test-slave-j368"
server_key: "7dcc3242f5dcd0b85c5e3d92c9402e70"
server_key_iv: "bfa5a0a07983ea07551c60cf388aefdc"
server_key_version: "encryptionkey@0"
I20260131 03:03:14.227950 15813 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:14.244503 15813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:14.248381 15813 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:14.248644 15813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:14.249351 15813 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:14.250347 15813 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:14.250430 15813 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:14.250506 15813 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:14.250553 15813 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:14.260587 15813 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:39595
I20260131 03:03:14.260628 15941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:39595 every 8 connection(s)
I20260131 03:03:14.261791 15813 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:03:14.266587 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 15813
I20260131 03:03:14.267140 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:36913
--local_ip_for_outbound_sockets=127.6.22.67
--tserver_master_addrs=127.6.22.126:44187
--webserver_port=42743
--webserver_interface=127.6.22.67
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.22.67
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/oryx_127.6.22.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
W20260131 03:03:14.274474 15811 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:14.263818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:39199 (local address 127.6.22.126:44187)
0131 03:03:14.263983 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:14.263987 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:14.264814 (+ 827us) server_negotiation.cc:408] Connection header received
0131 03:03:14.265761 (+ 947us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:14.265765 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:14.265837 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:14.265946 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:14.268416 (+ 2470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.268990 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.270008 (+ 1018us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.270158 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.274354 (+ 4196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:39199: BlockingRecv error: recv got EOF from 127.6.22.66:39199 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260131 03:03:14.275213 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:14.264133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:39199)
0131 03:03:14.264606 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:14.264644 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:14.265524 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:14.266137 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:14.266146 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:14.266772 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:14.268174 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.268202 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.269146 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.269154 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:14.269847 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.269863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.270148 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.270859 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:14.270907 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:14.274178 (+ 3271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":121,"threads_started":1}
W20260131 03:03:14.275728 15942 heartbeater.cc:646] Failed to heartbeat to 127.6.22.126:44187 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.22.126:44187: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:14.281631 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:14.276256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:35741)
0131 03:03:14.276474 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:14.276491 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:14.276589 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:14.276903 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:14.276907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:14.277108 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:14.277311 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.277317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.278082 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.278091 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:14.278477 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.278485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.278628 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.279266 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:14.279280 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:14.281458 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115}
W20260131 03:03:14.281757 15811 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:14.276366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:35741 (local address 127.6.22.126:44187)
0131 03:03:14.276519 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:14.276527 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:14.276545 (+ 18us) server_negotiation.cc:408] Connection header received
0131 03:03:14.276732 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:14.276736 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:14.276793 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:14.276884 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:14.277440 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.277951 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.278605 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.278805 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.281623 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:35741: BlockingRecv error: recv got EOF from 127.6.22.66:35741 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:14.287627 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:14.282384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:41701)
0131 03:03:14.282491 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:14.282507 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:14.282600 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:14.282929 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:14.282932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:14.283082 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:14.283259 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.283265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.284334 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.284337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:14.284790 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.284798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.284907 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.285458 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:14.285473 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:14.287507 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260131 03:03:14.287842 15811 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:14.282454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:41701 (local address 127.6.22.126:44187)
0131 03:03:14.282640 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:14.282646 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:14.282663 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:03:14.282736 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:14.282739 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:14.282794 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:14.282892 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:14.283403 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.284197 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.284908 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.285094 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.287720 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:41701: BlockingRecv error: recv got EOF from 127.6.22.66:41701 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260131 03:03:14.288009 15942 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260131 03:03:14.385324 15945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:14.385612 15945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:14.385699 15945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:14.389354 15945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:14.389467 15945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:14.389582 15945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:03:14.394430 15945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/krb5kdc/oryx_127.6.22.67.keytab
--principal=oryx/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:36913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=42743
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:14.395723 15945 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:14.396744 15945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:14.403949 15951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:14.403987 15953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:14.403995 15950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:14.404301 15945 server_base.cc:1060] running on GCE node
I20260131 03:03:14.404776 15945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:14.405441 15945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:14.406627 15945 hybrid_clock.cc:648] HybridClock initialized: now 1769828594406612 us; error 36 us; skew 500 ppm
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828594, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:14.409420 15945 init.cc:377] Logged in from keytab as oryx/127.6.22.67@KRBTEST.COM (short username oryx)
I20260131 03:03:14.410759 15945 webserver.cc:492] Webserver started at http://127.6.22.67:42743/ using document root <none> and password file <none>
I20260131 03:03:14.411445 15945 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:14.411549 15945 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:14.415342 15945 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260131 03:03:14.417410 15960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:14.418547 15945 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260131 03:03:14.418707 15945 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "1431943f8b194571a1f560019f5d8c53"
format_stamp: "Formatted at 2026-01-31 03:03:13 on dist-test-slave-j368"
server_key: "e0b6068ed716b242ac282917e4a97d12"
server_key_iv: "56860f4a1a35ead5f23bc2b9acbefd54"
server_key_version: "encryptionkey@0"
I20260131 03:03:14.419190 15945 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:14.432962 15945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:14.435897 15945 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:14.436159 15945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:14.436815 15945 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:14.437788 15945 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:14.437876 15945 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:14.437953 15945 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:14.437999 15945 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:14.448719 15945 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:36913
I20260131 03:03:14.448732 16073 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:36913 every 8 connection(s)
I20260131 03:03:14.449828 15945 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:03:14.455060 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 15945
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
W20260131 03:03:14.462314 15811 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:14.452494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:45261 (local address 127.6.22.126:44187)
0131 03:03:14.452795 (+ 301us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:14.452800 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:14.453833 (+ 1033us) server_negotiation.cc:408] Connection header received
0131 03:03:14.454827 (+ 994us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:14.454833 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:14.454909 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:14.455045 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:14.457333 (+ 2288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.457910 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.458690 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.458882 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.462176 (+ 3294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:45261: BlockingRecv error: recv got EOF from 127.6.22.67:45261 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
I20260131 03:03:14.462812 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:14.453003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:45261)
0131 03:03:14.453642 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:14.453690 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:14.454543 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:14.455267 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:14.455277 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:14.455866 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:14.457150 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.457169 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.458037 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.458041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:14.458557 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.458565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.458761 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.459428 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:14.459456 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:14.461995 (+ 2539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":161,"threads_started":1}
W20260131 03:03:14.463353 16074 heartbeater.cc:646] Failed to heartbeat to 127.6.22.126:44187 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.22.126:44187: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
I20260131 03:03:14.465196 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:14.456110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.6.22.126:44187)
0131 03:03:14.456420 (+ 310us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:14.456426 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:14.456443 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:03:14.456490 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:14.456495 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:14.456536 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:14.456644 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:14.457565 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.458176 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.458983 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.459168 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.460365 (+ 1197us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:14.460388 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:14.460401 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:14.460484 (+ 83us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:14.462741 (+ 2257us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:14.463327 (+ 586us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:14.463333 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:14.463338 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:14.463455 (+ 117us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:14.463732 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:14.463739 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:14.463741 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:14.464174 (+ 433us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:14.464391 (+ 217us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:14.464734 (+ 343us) server_negotiation.cc:300] Negotiation successful
0131 03:03:14.464983 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":110,"threads_started":1}
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:14.469449 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:14.463927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:34453)
0131 03:03:14.464120 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:14.464136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:14.464232 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:14.464525 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:14.464529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:14.464730 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:14.464989 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.464998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.465828 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.465832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:14.466362 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.466371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.466470 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.467071 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:14.467088 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:14.469292 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107}
W20260131 03:03:14.469547 15811 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:14.464008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:34453 (local address 127.6.22.126:44187)
0131 03:03:14.464144 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:14.464147 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:14.464180 (+ 33us) server_negotiation.cc:408] Connection header received
0131 03:03:14.464327 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:14.464331 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:14.464378 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:14.464467 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:14.465105 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.465711 (+ 606us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.466483 (+ 772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.466658 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.469429 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:34453: BlockingRecv error: recv got EOF from 127.6.22.67:34453 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:14 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:14.474946 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:14.470027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:54787)
0131 03:03:14.470165 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:14.470182 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:14.470266 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:14.470761 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:14.470763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:14.470909 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:14.471106 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.471112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.471912 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.471915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:14.472290 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:14.472298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.472400 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.473002 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:14.473016 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:14.474828 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260131 03:03:14.475162 15811 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:14.470110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:54787 (local address 127.6.22.126:44187)
0131 03:03:14.470246 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:14.470249 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:14.470263 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:03:14.470467 (+ 204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:14.470473 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:14.470583 (+ 110us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:14.470723 (+ 140us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:14.471266 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.471786 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:14.472399 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:14.472530 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:14.475009 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:54787: BlockingRecv error: recv got EOF from 127.6.22.67:54787 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260131 03:03:14.475236 16074 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jan 31 03:03:15 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:15 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:15.105005 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:15.098005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:36621)
0131 03:03:15.098352 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:15.098371 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:15.098494 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:15.099164 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:15.099170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:15.099451 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:15.099828 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:15.099840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:15.100915 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:15.100919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:15.101527 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:15.101537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:15.101670 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:15.102393 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:15.102408 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:15.104814 (+ 2406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":120,"threads_started":1}
W20260131 03:03:15.105104 16080 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:15.098328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:36621 (local address 127.6.22.126:44187)
0131 03:03:15.098667 (+ 339us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:15.098673 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:15.098696 (+ 23us) server_negotiation.cc:408] Connection header received
0131 03:03:15.098795 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:15.098799 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:15.098892 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:15.099077 (+ 185us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:15.100075 (+ 998us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:15.100776 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:15.101645 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:15.101880 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:15.104961 (+ 3081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:36621: BlockingRecv error: recv got EOF from 127.6.22.65:36621 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":121,"threads_started":1}
Jan 31 03:03:15 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:15 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:15.294200 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:15.288625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:42039)
0131 03:03:15.289008 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:15.289027 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:15.289175 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:15.289494 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:15.289498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:15.289720 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:15.289947 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:15.289954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:15.290737 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:15.290741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:15.291186 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:15.291194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:15.291341 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:15.292045 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:15.292059 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:15.293984 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
W20260131 03:03:15.294260 16080 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:15.288904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:42039 (local address 127.6.22.126:44187)
0131 03:03:15.289066 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:15.289073 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:15.289097 (+ 24us) server_negotiation.cc:408] Connection header received
0131 03:03:15.289302 (+ 205us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:15.289316 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:15.289369 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:15.289463 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:15.290091 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:15.290612 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:15.291321 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:15.291528 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:15.294139 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:42039: BlockingRecv error: recv got EOF from 127.6.22.66:42039 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jan 31 03:03:15 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:15 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:15.485266 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:15.476146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:51137)
0131 03:03:15.476444 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:15.476462 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:15.476573 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:15.476865 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:15.476868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:15.477156 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:15.477381 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:15.477390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:15.478250 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:15.478258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:15.478970 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:15.478980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:15.479139 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:15.482520 (+ 3381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:15.482544 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:15.485083 (+ 2539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":112,"threads_started":1}
W20260131 03:03:15.485442 16080 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:15.476260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:51137 (local address 127.6.22.126:44187)
0131 03:03:15.476426 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:15.476430 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:15.476540 (+ 110us) server_negotiation.cc:408] Connection header received
0131 03:03:15.476684 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:15.476687 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:15.476740 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:15.476824 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:15.477542 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:15.478102 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:15.479118 (+ 1016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:15.479339 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:15.485288 (+ 5949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:51137: BlockingRecv error: recv got EOF from 127.6.22.67:51137 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jan 31 03:03:16 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:16 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:16.111876 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:16.106106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:33089)
0131 03:03:16.106549 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:16.106571 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:16.106733 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:16.107100 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:16.107104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:16.107373 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:16.107638 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:16.107645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:16.108529 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:16.108534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:16.109016 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:16.109027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:16.109172 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:16.109828 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:16.109844 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:16.111742 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":85,"threads_started":1}
W20260131 03:03:16.112107 16084 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:16.106283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:33089 (local address 127.6.22.126:44187)
0131 03:03:16.106720 (+ 437us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:16.106725 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:16.106743 (+ 18us) server_negotiation.cc:408] Connection header received
0131 03:03:16.106842 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:16.106845 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:16.106905 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:16.107006 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:16.107780 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:16.108316 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:16.109151 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:16.109296 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:16.111951 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:33089: BlockingRecv error: recv got EOF from 127.6.22.65:33089 (error 108)
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":181,"threads_started":1}
Jan 31 03:03:16 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:16 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:16.300830 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:16.295414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:36079)
0131 03:03:16.295784 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:16.295801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:16.295957 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:16.296238 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:16.296241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:16.296453 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:16.296644 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:16.296706 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:16.297471 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:16.297477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:16.297857 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:16.297864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:16.298011 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:16.298615 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:16.298628 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:16.300609 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":99,"threads_started":1}
W20260131 03:03:16.301107 16084 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:16.295573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:36079 (local address 127.6.22.126:44187)
0131 03:03:16.295770 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:16.295776 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:16.295890 (+ 114us) server_negotiation.cc:408] Connection header received
0131 03:03:16.296063 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:16.296067 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:16.296123 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:16.296215 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:16.296835 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:16.297345 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:16.297997 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:16.298198 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:16.300804 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:36079: BlockingRecv error: recv got EOF from 127.6.22.66:36079 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jan 31 03:03:16 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:16 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:16.492391 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:16.486406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:59917)
0131 03:03:16.486837 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:16.486851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:16.486997 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:16.487442 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:16.487447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:16.487720 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:16.487947 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:16.487954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:16.488785 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:16.488795 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:16.489233 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:16.489240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:16.489391 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:16.490118 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:16.490139 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:16.492251 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":162,"threads_started":1}
W20260131 03:03:16.492642 16084 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:16.486592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:59917 (local address 127.6.22.126:44187)
0131 03:03:16.486778 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:16.486783 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:16.486937 (+ 154us) server_negotiation.cc:408] Connection header received
0131 03:03:16.487112 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:16.487117 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:16.487262 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:16.487410 (+ 148us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:16.488090 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:16.488626 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:16.489369 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:16.489556 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:16.492517 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:59917: BlockingRecv error: recv got EOF from 127.6.22.67:59917 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jan 31 03:03:17 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:17 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:17.119207 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:17.113004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:59739)
0131 03:03:17.113369 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:17.113403 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:17.113580 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:17.114085 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:17.114088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:17.114371 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:17.114598 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:17.114604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:17.115442 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:17.115447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:17.115836 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:17.115842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:17.115970 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:17.116742 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:17.116766 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:17.119059 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
W20260131 03:03:17.119499 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:17.113164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:59739 (local address 127.6.22.126:44187)
0131 03:03:17.113571 (+ 407us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:17.113574 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:17.113589 (+ 15us) server_negotiation.cc:408] Connection header received
0131 03:03:17.113708 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:17.113714 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:17.113897 (+ 183us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:17.114068 (+ 171us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:17.114740 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:17.115318 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:17.115947 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:17.116166 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:17.119319 (+ 3153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:59739: BlockingRecv error: recv got EOF from 127.6.22.65:59739 (error 108)
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":144,"threads_started":1}
Jan 31 03:03:17 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:17 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:17.307183 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:17.301780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:46161)
0131 03:03:17.302161 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:17.302179 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:17.302319 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:17.302654 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:17.302656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:17.302850 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:17.303062 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:17.303068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:17.303875 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:17.303877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:17.304279 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:17.304286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:17.304424 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:17.305172 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:17.305186 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:17.307047 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":155,"threads_started":1}
W20260131 03:03:17.307473 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:17.301874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:46161 (local address 127.6.22.126:44187)
0131 03:03:17.302093 (+ 219us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:17.302106 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:17.302304 (+ 198us) server_negotiation.cc:408] Connection header received
0131 03:03:17.302425 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:17.302429 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:17.302505 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:17.302600 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:17.303201 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:17.303766 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:17.304421 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:17.304573 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:17.307308 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:46161: BlockingRecv error: recv got EOF from 127.6.22.66:46161 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jan 31 03:03:17 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:17 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:17.498724 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:17.493384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:35105)
0131 03:03:17.493758 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:17.493775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:17.493936 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:17.494212 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:17.494215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:17.494439 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:17.494645 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:17.494651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:17.495411 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:17.495415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:17.495799 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:17.495806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:17.495931 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:17.496614 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:17.496628 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:17.498554 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":120,"threads_started":1}
W20260131 03:03:17.498831 16088 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:17.493566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:35105 (local address 127.6.22.126:44187)
0131 03:03:17.493745 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:17.493750 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:17.493868 (+ 118us) server_negotiation.cc:408] Connection header received
0131 03:03:17.494042 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:17.494047 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:17.494098 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:17.494191 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:17.494770 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:17.495293 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:17.495921 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:17.496113 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:17.498699 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:35105: BlockingRecv error: recv got EOF from 127.6.22.67:35105 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jan 31 03:03:18 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:18 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:18.125700 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:18.120108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:55669)
0131 03:03:18.120695 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:18.120710 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:18.120813 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:18.121215 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:18.121218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:18.121411 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:18.121633 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:18.121640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:18.122529 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:18.122532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:18.122942 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:18.122948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:18.123050 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:18.123727 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:18.123741 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:18.125572 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":133,"threads_started":1}
W20260131 03:03:18.125998 16092 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:18.120401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:55669 (local address 127.6.22.126:44187)
0131 03:03:18.120861 (+ 460us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:18.120866 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:18.120886 (+ 20us) server_negotiation.cc:408] Connection header received
0131 03:03:18.120943 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:18.120946 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:18.121006 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:18.121151 (+ 145us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:18.121798 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:18.122410 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:18.123077 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:18.123230 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:18.125818 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:55669: BlockingRecv error: recv got EOF from 127.6.22.65:55669 (error 108)
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":77,"threads_started":1}
Jan 31 03:03:18 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:18 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:18.313416 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:18.308007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:40965)
0131 03:03:18.308379 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:18.308398 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:18.308553 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:18.308862 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:18.308866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:18.309083 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:18.309283 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:18.309290 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:18.310066 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:18.310072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:18.310465 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:18.310473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:18.310606 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:18.311272 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:18.311286 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:18.313238 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":131,"threads_started":1}
W20260131 03:03:18.313488 16092 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:18.308184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:40965 (local address 127.6.22.126:44187)
0131 03:03:18.308361 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:18.308366 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:18.308487 (+ 121us) server_negotiation.cc:408] Connection header received
0131 03:03:18.308693 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:18.308696 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:18.308743 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:18.308837 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:18.309413 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:18.309949 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:18.310583 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:18.310786 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:18.313365 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:40965: BlockingRecv error: recv got EOF from 127.6.22.66:40965 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jan 31 03:03:18 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:18 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:18.505654 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:18.499795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:43963)
0131 03:03:18.500163 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:18.500177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:18.500292 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:18.500603 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:18.500605 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:18.500827 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:18.501022 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:18.501027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:18.502190 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:18.502194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:18.502735 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:18.502745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:18.502860 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:18.503441 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:18.503464 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:18.505540 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":86,"threads_started":1}
W20260131 03:03:18.505899 16092 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:18.499958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:43963 (local address 127.6.22.126:44187)
0131 03:03:18.500163 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:18.500171 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:18.500272 (+ 101us) server_negotiation.cc:408] Connection header received
0131 03:03:18.500378 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:18.500385 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:18.500457 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:18.500580 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:18.501152 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:18.502060 (+ 908us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:18.502858 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:18.503109 (+ 251us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:18.505731 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:43963: BlockingRecv error: recv got EOF from 127.6.22.67:43963 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jan 31 03:03:19 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:19 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:19.131873 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:19.126618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:55629)
0131 03:03:19.126910 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:19.126947 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:19.127057 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:19.127447 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:19.127450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:19.127717 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:19.127990 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:19.128001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:19.128993 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:19.128999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:19.129433 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:19.129442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:19.129542 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:19.130056 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:19.130069 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:19.131752 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":105,"threads_started":1}
W20260131 03:03:19.132058 16096 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:19.126688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:55629 (local address 127.6.22.126:44187)
0131 03:03:19.127003 (+ 315us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:19.127007 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:19.127023 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:03:19.127198 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:19.127203 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:19.127280 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:19.127392 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:19.128163 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:19.128736 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:19.129543 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:19.129683 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:19.131907 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:55629: BlockingRecv error: recv got EOF from 127.6.22.65:55629 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":114,"threads_started":1}
Jan 31 03:03:19 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:19 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:19.319614 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:19.314370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:59559)
0131 03:03:19.314702 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:19.314717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:19.314810 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:19.315058 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:19.315061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:19.315279 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:19.315485 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:19.315492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:19.316260 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:19.316263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:19.316840 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:19.316850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:19.316952 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:19.317562 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:19.317575 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:19.319464 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":107,"threads_started":1}
W20260131 03:03:19.320004 16096 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:19.314576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:59559 (local address 127.6.22.126:44187)
0131 03:03:19.314739 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:19.314743 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:19.314761 (+ 18us) server_negotiation.cc:408] Connection header received
0131 03:03:19.314899 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:19.314901 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:19.314950 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:19.315021 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:19.315602 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:19.316107 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:19.316953 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:19.317092 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:19.319832 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:59559: BlockingRecv error: recv got EOF from 127.6.22.66:59559 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jan 31 03:03:19 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:19 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:19.512758 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:19.506931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:36371)
0131 03:03:19.507291 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:19.507305 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:19.507430 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:19.508054 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:19.508059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:19.508337 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:19.508567 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:19.508576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:19.509561 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:19.509565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:19.510046 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:19.510054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:19.510156 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:19.510692 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:19.510706 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:19.512578 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":140,"threads_started":1}
W20260131 03:03:19.512945 16096 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:19.507065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:36371 (local address 127.6.22.126:44187)
0131 03:03:19.507253 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:19.507257 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:19.507528 (+ 271us) server_negotiation.cc:408] Connection header received
0131 03:03:19.507613 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:19.507616 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:19.507686 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:19.507806 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:19.508813 (+ 1007us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:19.509421 (+ 608us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:19.510161 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:19.510297 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:19.512800 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:36371: BlockingRecv error: recv got EOF from 127.6.22.67:36371 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jan 31 03:03:20 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:20 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:20.138399 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:20.132980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:37481)
0131 03:03:20.133285 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:20.133300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:20.133399 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:20.133676 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:20.133679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:20.133857 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:20.134056 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:20.134061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:20.134852 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:20.134855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:20.135235 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:20.135242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:20.135343 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:20.135895 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:20.135909 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:20.138218 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
W20260131 03:03:20.138481 16100 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:20.133091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:37481 (local address 127.6.22.126:44187)
0131 03:03:20.133349 (+ 258us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:20.133353 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:20.133368 (+ 15us) server_negotiation.cc:408] Connection header received
0131 03:03:20.133505 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:20.133508 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:20.133556 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:20.133642 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:20.134164 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:20.134741 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:20.135350 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:20.135490 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:20.138358 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:37481: BlockingRecv error: recv got EOF from 127.6.22.65:37481 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":65,"threads_started":1}
Jan 31 03:03:20 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:20 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:20.326403 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:20.320654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:44395)
0131 03:03:20.321057 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:20.321072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:20.321201 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:20.321530 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:20.321534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:20.321758 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:20.322007 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:20.322013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:20.322829 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:20.322832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:20.323280 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:20.323288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:20.323408 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:20.324111 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:20.324124 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:20.326266 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":113,"threads_started":1}
W20260131 03:03:20.326691 16100 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:20.320793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:44395 (local address 127.6.22.126:44187)
0131 03:03:20.320943 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:20.320947 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:20.321158 (+ 211us) server_negotiation.cc:408] Connection header received
0131 03:03:20.321296 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:20.321299 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:20.321356 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:20.321449 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:20.322158 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:20.322690 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:20.323396 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:20.323541 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:20.326530 (+ 2989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:44395: BlockingRecv error: recv got EOF from 127.6.22.66:44395 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jan 31 03:03:20 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:20 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:20.523855 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:20.514060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:60707)
0131 03:03:20.514491 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:20.514506 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:20.514675 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:20.515059 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:20.515062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:20.515316 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:20.515776 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:20.515790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:20.516780 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:20.516784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:20.517201 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:20.517210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:20.517342 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:20.521109 (+ 3767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:20.521129 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:20.523693 (+ 2564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":130,"threads_started":1}
W20260131 03:03:20.524107 16100 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:20.514315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:60707 (local address 127.6.22.126:44187)
0131 03:03:20.514554 (+ 239us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:20.514559 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:20.514651 (+ 92us) server_negotiation.cc:408] Connection header received
0131 03:03:20.514799 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:20.514805 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:20.514874 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:20.514979 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:20.515952 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:20.516509 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:20.517340 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:20.517480 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:20.523927 (+ 6447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:60707: BlockingRecv error: recv got EOF from 127.6.22.67:60707 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Jan 31 03:03:21 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:21 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:21.145226 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:21.139355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:43945)
0131 03:03:21.139940 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:21.139957 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:21.140088 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:21.140388 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:21.140391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:21.140642 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:21.140924 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:21.140930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:21.141708 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:21.141711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:21.142134 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:21.142141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:21.142259 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:21.142908 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:21.142922 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:21.145048 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":96,"threads_started":1}
W20260131 03:03:21.145443 16104 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:21.139493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:43945 (local address 127.6.22.126:44187)
0131 03:03:21.139779 (+ 286us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:21.139783 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:21.140051 (+ 268us) server_negotiation.cc:408] Connection header received
0131 03:03:21.140189 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:21.140192 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:21.140246 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:21.140340 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:21.141036 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:21.141564 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:21.142249 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:21.142392 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:21.145242 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:43945: BlockingRecv error: recv got EOF from 127.6.22.65:43945 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":62,"threads_started":1}
Jan 31 03:03:21 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:21 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:21.333516 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:21.327392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:38879)
0131 03:03:21.327780 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:21.327799 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:21.327945 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:21.328342 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:21.328348 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:21.328628 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:21.329016 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:21.329025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:21.329803 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:21.329806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:21.330250 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:21.330256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:21.330356 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:21.331032 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:21.331045 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:21.333382 (+ 2337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
W20260131 03:03:21.333698 16104 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:21.327673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:38879 (local address 127.6.22.126:44187)
0131 03:03:21.327860 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:21.327866 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:21.327883 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:03:21.328101 (+ 218us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:21.328104 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:21.328165 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:21.328289 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:21.329148 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:21.329679 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:21.330358 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:21.330500 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:21.333528 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:38879: BlockingRecv error: recv got EOF from 127.6.22.66:38879 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jan 31 03:03:21 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:21 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:21.530637 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:21.524805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:41023)
0131 03:03:21.525145 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:21.525160 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:21.525284 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:21.525565 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:21.525568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:21.525793 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:21.526022 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:21.526028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:21.526862 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:21.526865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:21.527279 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:21.527286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:21.527385 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:21.528143 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:21.528161 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:21.530453 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":122,"threads_started":1}
W20260131 03:03:21.530843 16104 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:21.524896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:41023 (local address 127.6.22.126:44187)
0131 03:03:21.525057 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:21.525061 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:21.525227 (+ 166us) server_negotiation.cc:408] Connection header received
0131 03:03:21.525352 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:21.525354 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:21.525407 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:21.525482 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:21.526162 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:21.526734 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:21.527525 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:21.527853 (+ 328us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:21.530670 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:41023: BlockingRecv error: recv got EOF from 127.6.22.67:41023 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jan 31 03:03:22 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:22 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:22.152413 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:22.146381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:58765)
0131 03:03:22.146811 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:22.146829 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:22.146961 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:22.147374 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:22.147376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:22.147586 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:22.147812 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:22.147818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:22.148844 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:22.148848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:22.149256 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:22.149264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:22.149384 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:22.150014 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:22.150032 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:22.152212 (+ 2180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":117,"threads_started":1}
W20260131 03:03:22.152472 16108 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:22.146750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:58765 (local address 127.6.22.126:44187)
0131 03:03:22.147054 (+ 304us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:22.147059 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:22.147077 (+ 18us) server_negotiation.cc:408] Connection header received
0131 03:03:22.147135 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:22.147138 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:22.147197 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:22.147308 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:22.148079 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:22.148699 (+ 620us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:22.149385 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:22.149536 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:22.152348 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:58765: BlockingRecv error: recv got EOF from 127.6.22.65:58765 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
Jan 31 03:03:22 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:22 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:22.339962 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:22.334516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:55447)
0131 03:03:22.334874 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:22.334889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:22.335043 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:22.335336 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:22.335339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:22.335565 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:22.335771 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:22.335778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:22.336739 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:22.336744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:22.337139 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:22.337147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:22.337278 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:22.337864 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:22.337877 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:22.339776 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":147,"threads_started":1}
W20260131 03:03:22.340055 16108 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:22.334602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:55447 (local address 127.6.22.126:44187)
0131 03:03:22.334779 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:22.334785 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:22.334982 (+ 197us) server_negotiation.cc:408] Connection header received
0131 03:03:22.335155 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:22.335159 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:22.335218 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:22.335311 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:22.335907 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:22.336583 (+ 676us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:22.337271 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:22.337476 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:22.339930 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:55447: BlockingRecv error: recv got EOF from 127.6.22.66:55447 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jan 31 03:03:22 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:22 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:22.536983 16110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:22.531608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:51995)
0131 03:03:22.532138 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:22.532155 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:22.532285 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:22.532613 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:22.532616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:22.532827 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:22.533034 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:22.533040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:22.533817 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:22.533823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:22.534213 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:22.534220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:22.534355 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:22.534928 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:22.534941 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:22.536801 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":97,"threads_started":1}
W20260131 03:03:22.537053 16108 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:22.531642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:51995 (local address 127.6.22.126:44187)
0131 03:03:22.531821 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:22.531826 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:22.532241 (+ 415us) server_negotiation.cc:408] Connection header received
0131 03:03:22.532405 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:22.532410 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:22.532476 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:22.532570 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:22.533169 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:22.533688 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:22.534335 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:22.534527 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:22.536942 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:51995: BlockingRecv error: recv got EOF from 127.6.22.67:51995 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jan 31 03:03:23 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:23 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:23.158720 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:23.153461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:42023)
0131 03:03:23.153768 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:23.153783 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:23.153882 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:23.154152 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:23.154154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:23.154341 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:23.154543 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:23.154550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:23.155303 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:23.155306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:23.155696 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:23.155703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:23.155824 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:23.156380 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:23.156393 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:23.158575 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":99,"threads_started":1}
W20260131 03:03:23.159121 16112 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:23.153551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:42023 (local address 127.6.22.126:44187)
0131 03:03:23.153820 (+ 269us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:23.153824 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:23.153839 (+ 15us) server_negotiation.cc:408] Connection header received
0131 03:03:23.153983 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:23.153986 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:23.154033 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:23.154117 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:23.154675 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:23.155196 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:23.155862 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:23.156000 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:23.158932 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:42023: BlockingRecv error: recv got EOF from 127.6.22.65:42023 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":67,"threads_started":1}
Jan 31 03:03:23 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:23 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:23.347429 16113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:23.341129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:57677)
0131 03:03:23.341605 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:23.341637 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:23.341821 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:23.342181 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:23.342184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:23.342421 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:23.342687 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:23.342694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:23.343510 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:23.343513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:23.343961 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:23.343968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:23.344085 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:23.344821 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:23.344835 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:23.347232 (+ 2397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":98,"threads_started":1}
W20260131 03:03:23.347501 16112 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:23.341410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:57677 (local address 127.6.22.126:44187)
0131 03:03:23.341639 (+ 229us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:23.341645 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:23.341762 (+ 117us) server_negotiation.cc:408] Connection header received
0131 03:03:23.341936 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:23.341939 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:23.342012 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:23.342113 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:23.342820 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:23.343340 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:23.344084 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:23.344237 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:23.347377 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:57677: BlockingRecv error: recv got EOF from 127.6.22.66:57677 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jan 31 03:03:23 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:23 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:23.544117 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:23.537860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:42489)
0131 03:03:23.538331 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:23.538353 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:23.538668 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:23.539071 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:23.539074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:23.539335 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:23.539569 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:23.539575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:23.540374 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:23.540378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:23.540839 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:23.540846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:23.540984 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:23.541774 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:23.541789 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:23.543903 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":108,"threads_started":1}
W20260131 03:03:23.544178 16112 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:23.538122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:42489 (local address 127.6.22.126:44187)
0131 03:03:23.538331 (+ 209us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:23.538338 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:23.538565 (+ 227us) server_negotiation.cc:408] Connection header received
0131 03:03:23.538862 (+ 297us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:23.538867 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:23.538944 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:23.539036 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:23.539718 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:23.540239 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:23.540953 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:23.541138 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:23.544060 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:42489: BlockingRecv error: recv got EOF from 127.6.22.67:42489 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jan 31 03:03:24 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:24 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:24.166347 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:24.159995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:35005)
0131 03:03:24.160471 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:24.160488 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:24.160623 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:24.161059 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:24.161062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:24.161365 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:24.161614 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:24.161620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:24.162529 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:24.162533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:24.163123 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:24.163131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:24.163269 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:24.164019 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:24.164032 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:24.166205 (+ 2173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":127,"threads_started":1}
W20260131 03:03:24.166658 16116 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:24.160164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:35005 (local address 127.6.22.126:44187)
0131 03:03:24.160499 (+ 335us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:24.160503 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:24.160589 (+ 86us) server_negotiation.cc:408] Connection header received
0131 03:03:24.160829 (+ 240us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:24.160833 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:24.160891 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:24.160991 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:24.161763 (+ 772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:24.162364 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:24.163245 (+ 881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:24.163450 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:24.166359 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:35005: BlockingRecv error: recv got EOF from 127.6.22.65:35005 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
Jan 31 03:03:24 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:24 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:24.354898 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:24.348655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:48177)
0131 03:03:24.349206 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:24.349220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:24.349377 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:24.349691 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:24.349694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:24.349941 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:24.350160 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:24.350167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:24.351057 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:24.351061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:24.351461 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:24.351467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:24.351605 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:24.352343 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:24.352365 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:24.354747 (+ 2382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":99,"threads_started":1}
W20260131 03:03:24.355284 16116 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:24.348749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:48177 (local address 127.6.22.126:44187)
0131 03:03:24.348925 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:24.348933 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:24.349308 (+ 375us) server_negotiation.cc:408] Connection header received
0131 03:03:24.349499 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:24.349502 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:24.349558 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:24.349651 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:24.350282 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:24.350945 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:24.351581 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:24.351772 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:24.354995 (+ 3223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:48177: BlockingRecv error: recv got EOF from 127.6.22.66:48177 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jan 31 03:03:24 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:24 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:24.550729 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:24.545303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:45511)
0131 03:03:24.545802 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:24.545817 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:24.545949 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:24.546253 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:24.546256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:24.546458 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:24.546678 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:24.546685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:24.547511 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:24.547514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:24.547933 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:24.547940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:24.548043 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:24.548719 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:24.548743 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:24.550590 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":143,"threads_started":1}
W20260131 03:03:24.551017 16116 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:24.545432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:45511 (local address 127.6.22.126:44187)
0131 03:03:24.545673 (+ 241us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:24.545678 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:24.545922 (+ 244us) server_negotiation.cc:408] Connection header received
0131 03:03:24.546065 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:24.546068 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:24.546126 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:24.546204 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:24.546831 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:24.547389 (+ 558us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:24.548183 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:24.548370 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:24.550871 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:45511: BlockingRecv error: recv got EOF from 127.6.22.67:45511 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jan 31 03:03:25 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:25 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:25.174396 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:25.167605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:42747)
0131 03:03:25.168085 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:25.168104 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:25.168263 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:25.168708 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:25.168712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:25.168971 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:25.169316 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:25.169324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:25.170395 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:25.170400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:25.170861 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:25.170871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:25.171002 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:25.171746 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:25.171762 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:25.174257 (+ 2495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":212,"threads_started":1}
W20260131 03:03:25.174710 16120 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:25.167778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:42747 (local address 127.6.22.126:44187)
0131 03:03:25.168267 (+ 489us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:25.168272 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:25.168299 (+ 27us) server_negotiation.cc:408] Connection header received
0131 03:03:25.168346 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:25.168349 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:25.168401 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:25.168568 (+ 167us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:25.169513 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:25.170209 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:25.171001 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:25.171148 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:25.174442 (+ 3294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:42747: BlockingRecv error: recv got EOF from 127.6.22.65:42747 (error 108)
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":119,"threads_started":1}
Jan 31 03:03:25 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:25 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:25.361305 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:25.356035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:37161)
0131 03:03:25.356340 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:25.356354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:25.356471 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:25.356775 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:25.356784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:25.356970 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:25.357187 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:25.357193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:25.357938 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:25.357941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:25.358335 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:25.358342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:25.358459 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:25.359053 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:25.359068 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:25.361132 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
W20260131 03:03:25.361601 16120 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:25.356131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:37161 (local address 127.6.22.126:44187)
0131 03:03:25.356338 (+ 207us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:25.356343 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:25.356437 (+ 94us) server_negotiation.cc:408] Connection header received
0131 03:03:25.356564 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:25.356567 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:25.356621 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:25.356720 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:25.357310 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:25.357828 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:25.358466 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:25.358609 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:25.361313 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:37161: BlockingRecv error: recv got EOF from 127.6.22.66:37161 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jan 31 03:03:25 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:25 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:25.561870 16122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:25.551984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:36773)
0131 03:03:25.552392 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:25.552408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:25.552577 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:25.552897 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:25.552901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:25.553112 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:25.553305 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:25.553311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:25.554072 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:25.554078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:25.554460 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:25.554467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:25.554603 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:25.558691 (+ 4088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:25.558714 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:25.561676 (+ 2962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":144,"threads_started":1}
W20260131 03:03:25.561996 16120 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:25.552082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:36773 (local address 127.6.22.126:44187)
0131 03:03:25.552316 (+ 234us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:25.552323 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:25.552504 (+ 181us) server_negotiation.cc:408] Connection header received
0131 03:03:25.552711 (+ 207us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:25.552716 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:25.552774 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:25.552869 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:25.553447 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:25.553951 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:25.554598 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:25.554790 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:25.561853 (+ 7063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:36773: BlockingRecv error: recv got EOF from 127.6.22.67:36773 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jan 31 03:03:26 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:26 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:26.181396 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:26.175689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:53569)
0131 03:03:26.176101 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:26.176117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:26.176241 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:26.176534 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:26.176537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:26.176774 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:26.176982 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:26.176988 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:26.177757 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:26.177759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:26.178212 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:26.178221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:26.178344 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:26.179044 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:26.179061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:26.181266 (+ 2205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":105,"threads_started":1}
W20260131 03:03:26.181604 16124 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:26.175857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:53569 (local address 127.6.22.126:44187)
0131 03:03:26.176124 (+ 267us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:26.176128 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:26.176204 (+ 76us) server_negotiation.cc:408] Connection header received
0131 03:03:26.176328 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:26.176331 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:26.176381 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:26.176464 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:26.177124 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:26.177647 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:26.178330 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:26.178527 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:26.181420 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:53569: BlockingRecv error: recv got EOF from 127.6.22.65:53569 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":66,"threads_started":1}
Jan 31 03:03:26 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:26 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:26.369059 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:26.362442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:36841)
0131 03:03:26.362832 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:26.362853 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:26.363013 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:26.363355 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:26.363359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:26.363607 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:26.363840 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:26.363847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:26.364804 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:26.364808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:26.365347 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:26.365357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:26.365500 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:26.366231 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:26.366244 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:26.368912 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":180,"threads_started":1}
W20260131 03:03:26.369385 16124 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:26.362639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:36841 (local address 127.6.22.126:44187)
0131 03:03:26.362825 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:26.362829 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:26.362985 (+ 156us) server_negotiation.cc:408] Connection header received
0131 03:03:26.363115 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:26.363119 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:26.363182 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:26.363276 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:26.363996 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:26.364628 (+ 632us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:26.365474 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:26.365695 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:26.369229 (+ 3534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:36841: BlockingRecv error: recv got EOF from 127.6.22.66:36841 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jan 31 03:03:26 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:26 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:26.568809 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:26.563023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:46555)
0131 03:03:26.563491 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:26.563508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:26.563634 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:26.563926 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:26.563928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:26.564123 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:26.564336 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:26.564341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:26.565263 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:26.565266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:26.565687 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:26.565694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:26.565801 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:26.566445 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:26.566459 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:26.568575 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":142,"threads_started":1}
W20260131 03:03:26.569051 16124 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:26.563095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:46555 (local address 127.6.22.126:44187)
0131 03:03:26.563287 (+ 192us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:26.563292 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:26.563601 (+ 309us) server_negotiation.cc:408] Connection header received
0131 03:03:26.563734 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:26.563737 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:26.563803 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:26.563883 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:26.564477 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:26.565141 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:26.565823 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:26.565977 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:26.568880 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:46555: BlockingRecv error: recv got EOF from 127.6.22.67:46555 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jan 31 03:03:27 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:27 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:27.188035 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:27.182212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:37945)
0131 03:03:27.182737 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:27.182755 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:27.182912 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:27.183228 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:27.183230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:27.183567 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:27.183829 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:27.183836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:27.184639 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:27.184642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:27.185100 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:27.185107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:27.185210 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:27.185844 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:27.185858 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:27.187866 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":137,"threads_started":1}
W20260131 03:03:27.188242 16128 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:27.182410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:37945 (local address 127.6.22.126:44187)
0131 03:03:27.182655 (+ 245us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:27.182659 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:27.182857 (+ 198us) server_negotiation.cc:408] Connection header received
0131 03:03:27.183017 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:27.183019 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:27.183078 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:27.183209 (+ 131us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:27.183962 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:27.184516 (+ 554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:27.185209 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:27.185367 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:27.188072 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:37945: BlockingRecv error: recv got EOF from 127.6.22.65:37945 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
Jan 31 03:03:27 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:27 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:27.375720 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:27.370126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:52653)
0131 03:03:27.370522 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:27.370540 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:27.370723 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:27.371043 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:27.371045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:27.371266 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:27.371470 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:27.371477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:27.372248 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:27.372253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:27.372679 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:27.372687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:27.372826 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:27.373400 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:27.373413 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:27.375502 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":149,"threads_started":1}
W20260131 03:03:27.375831 16128 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:27.370335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:52653 (local address 127.6.22.126:44187)
0131 03:03:27.370522 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:27.370529 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:27.370649 (+ 120us) server_negotiation.cc:408] Connection header received
0131 03:03:27.370842 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:27.370846 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:27.370913 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:27.371011 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:27.371619 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:27.372132 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:27.372803 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:27.373010 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:27.375676 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:52653: BlockingRecv error: recv got EOF from 127.6.22.66:52653 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jan 31 03:03:27 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:27 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:27.575311 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:27.569885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:50803)
0131 03:03:27.570184 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:27.570199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:27.570296 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:27.570548 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:27.570551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:27.570741 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:27.570956 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:27.570964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:27.571705 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:27.571708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:27.572094 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:27.572101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:27.572198 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:27.572898 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:27.572913 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:27.575139 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":121,"threads_started":1}
W20260131 03:03:27.575404 16128 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:27.570138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:50803 (local address 127.6.22.126:44187)
0131 03:03:27.570287 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:27.570291 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:27.570307 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:03:27.570374 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:27.570377 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:27.570428 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:27.570515 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:27.571093 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:27.571589 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:27.572305 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:27.572535 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:27.575286 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:50803: BlockingRecv error: recv got EOF from 127.6.22.67:50803 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jan 31 03:03:28 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:28 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:28.195967 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:28.189269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:47277)
0131 03:03:28.189611 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:28.189628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:28.189740 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:28.190060 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:28.190063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:28.190259 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:28.190470 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:28.190475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:28.191534 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:28.191540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:28.191986 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:28.191996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:28.192126 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:28.192987 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:28.193002 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:28.195746 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":134,"threads_started":1}
W20260131 03:03:28.196115 16132 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:28.189367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:47277 (local address 127.6.22.126:44187)
0131 03:03:28.189699 (+ 332us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:28.189703 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:28.189720 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:03:28.189835 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:28.189838 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:28.189900 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:28.189988 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:28.190688 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:28.191319 (+ 631us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:28.192189 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:28.192403 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:28.195950 (+ 3547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:47277: BlockingRecv error: recv got EOF from 127.6.22.65:47277 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":94,"threads_started":1}
Jan 31 03:03:28 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:28 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:28.381877 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:28.376638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:42803)
0131 03:03:28.377038 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:28.377053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:28.377164 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:28.377421 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:28.377423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:28.377605 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:28.377813 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:28.377820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:28.378701 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:28.378704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:28.379155 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:28.379164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:28.379263 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:28.379960 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:28.379974 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:28.381755 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":91,"threads_started":1}
W20260131 03:03:28.382018 16132 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:28.376923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:42803 (local address 127.6.22.126:44187)
0131 03:03:28.377074 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:28.377078 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:28.377110 (+ 32us) server_negotiation.cc:408] Connection header received
0131 03:03:28.377253 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:28.377256 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:28.377305 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:28.377378 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:28.377932 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:28.378573 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:28.379276 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:28.379463 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:28.381894 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:42803: BlockingRecv error: recv got EOF from 127.6.22.66:42803 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jan 31 03:03:28 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:28 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.67@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:28.582574 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:28.576244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.67:58147)
0131 03:03:28.576759 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:28.576781 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:28.576991 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:28.577318 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:28.577321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:28.577588 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:28.577820 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:28.577827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:28.578642 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:28.578648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:28.579059 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:28.579066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:28.579213 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:28.580033 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:28.580054 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:28.582309 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":157,"threads_started":1}
W20260131 03:03:28.582620 16132 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:28.576490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:58147 (local address 127.6.22.126:44187)
0131 03:03:28.576760 (+ 270us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:28.576776 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:28.576899 (+ 123us) server_negotiation.cc:408] Connection header received
0131 03:03:28.577111 (+ 212us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:28.577115 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:28.577184 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:28.577284 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:28.577952 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:28.578514 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:28.579181 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:28.579369 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:28.582501 (+ 3132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.67:58147: BlockingRecv error: recv got EOF from 127.6.22.67:58147 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jan 31 03:03:29 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:29 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.65@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:29.203382 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:29.197355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.65:57643)
0131 03:03:29.197668 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:29.197684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:29.197806 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:29.198134 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:29.198136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:29.198323 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:29.198538 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:29.198544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.199401 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:29.199407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:29.199916 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:29.199928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.200085 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:29.201260 (+ 1175us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:29.201276 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:29.203227 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
W20260131 03:03:29.203651 16136 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:29.197454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:57643 (local address 127.6.22.126:44187)
0131 03:03:29.197749 (+ 295us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:29.197759 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:29.197772 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:03:29.197940 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:29.197945 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:29.198006 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:29.198105 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:29.198651 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.199199 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:29.200094 (+ 895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.200255 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:29.203480 (+ 3225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.65:57643: BlockingRecv error: recv got EOF from 127.6.22.65:57643 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":66,"threads_started":1}
Jan 31 03:03:29 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:03:29 dist-test-slave-j368 krb5kdc[15069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.22.66@KRBTEST.COM for oryx/127.6.22.126@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:03:29.388993 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:29.383023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:44187 (local address 127.6.22.66:49417)
0131 03:03:29.383329 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:29.383347 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:29.383504 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:29.383977 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:29.383983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:29.384191 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:29.384409 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:29.384416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.385405 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:29.385409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:29.385804 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:29.385813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.385929 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:29.386565 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:29.386605 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:29.388860 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.22.126:44187: Server oryx/127.6.22.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":110,"threads_started":1}
W20260131 03:03:29.389245 16136 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:29.383153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:49417 (local address 127.6.22.126:44187)
0131 03:03:29.383324 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:29.383329 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:29.383582 (+ 253us) server_negotiation.cc:408] Connection header received
0131 03:03:29.383633 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:29.383637 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:29.383709 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:29.383786 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:29.384601 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.385244 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:29.385940 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.386089 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:29.389083 (+ 2994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.22.66:49417: BlockingRecv error: recv got EOF from 127.6.22.66:49417 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260131 03:03:29.466324 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 15661
I20260131 03:03:29.473616 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 15813
I20260131 03:03:29.480482 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 15945
I20260131 03:03:29.486538 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 15591
2026-01-31T03:03:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (19132 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-01-31T03:03:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:03:29Z Disabled control of system clock
I20260131 03:03:29.518916 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:38791
--webserver_interface=127.6.22.126
--webserver_port=0
--builtin_ntp_servers=127.6.22.84:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:38791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260131 03:03:29.632567 16141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:29.632885 16141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:29.632933 16141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:29.636405 16141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:03:29.636494 16141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:29.636512 16141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:29.636531 16141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:03:29.636549 16141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:03:29.640837 16141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:38791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:38791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:29.642038 16141 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:29.643200 16141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:29.650060 16147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:29.650051 16149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:29.650166 16141 server_base.cc:1060] running on GCE node
W20260131 03:03:29.650060 16146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:29.651051 16141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:29.652331 16141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:29.653563 16141 hybrid_clock.cc:648] HybridClock initialized: now 1769828609653531 us; error 59 us; skew 500 ppm
I20260131 03:03:29.655827 16141 webserver.cc:492] Webserver started at http://127.6.22.126:34939/ using document root <none> and password file <none>
I20260131 03:03:29.656486 16141 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:29.656575 16141 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:29.657001 16141 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:29.658959 16141 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "a42d47f6474a44fcbcb60900eec90d2e"
format_stamp: "Formatted at 2026-01-31 03:03:29 on dist-test-slave-j368"
server_key: "aa794eb24bd6600875be7ac4cd092923"
server_key_iv: "c2d0675f244c473342c315aea776b8cf"
server_key_version: "encryptionkey@0"
I20260131 03:03:29.659528 16141 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "a42d47f6474a44fcbcb60900eec90d2e"
format_stamp: "Formatted at 2026-01-31 03:03:29 on dist-test-slave-j368"
server_key: "aa794eb24bd6600875be7ac4cd092923"
server_key_iv: "c2d0675f244c473342c315aea776b8cf"
server_key_version: "encryptionkey@0"
I20260131 03:03:29.663718 16141 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260131 03:03:29.666991 16155 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:29.668439 16141 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260131 03:03:29.668684 16141 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "a42d47f6474a44fcbcb60900eec90d2e"
format_stamp: "Formatted at 2026-01-31 03:03:29 on dist-test-slave-j368"
server_key: "aa794eb24bd6600875be7ac4cd092923"
server_key_iv: "c2d0675f244c473342c315aea776b8cf"
server_key_version: "encryptionkey@0"
I20260131 03:03:29.668879 16141 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:29.687773 16141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:29.688606 16141 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:29.688911 16141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:29.696494 16141 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:38791
I20260131 03:03:29.696517 16207 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:38791 every 8 connection(s)
I20260131 03:03:29.697669 16141 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:03:29.700806 16208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:29.705721 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 16141
I20260131 03:03:29.705888 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:03:29.706166 6233 external_mini_cluster.cc:1468] Setting key 8053649861fc4a225f9450eee7230309
I20260131 03:03:29.707136 16208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e: Bootstrap starting.
I20260131 03:03:29.709637 16208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:29.710439 16208 log.cc:826] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e: Log is configured to *not* fsync() on all Append() calls
I20260131 03:03:29.712568 16208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e: No bootstrap required, opened a new log
I20260131 03:03:29.714520 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:29.707565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.6.22.126:38791)
0131 03:03:29.708285 (+ 720us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:29.708294 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:29.708332 (+ 38us) server_negotiation.cc:408] Connection header received
0131 03:03:29.709021 (+ 689us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:29.709045 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:29.709433 (+ 388us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:29.709800 (+ 367us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:29.710330 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.711138 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:29.712008 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.712338 (+ 330us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:29.712938 (+ 600us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:29.712960 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:29.712971 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:03:29.713360 (+ 389us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:29.713405 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:03:29.713421 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:29.713545 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:29.713680 (+ 135us) server_negotiation.cc:300] Negotiation successful
0131 03:03:29.713926 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":161,"threads_started":1}
I20260131 03:03:29.715415 16208 raft_consensus.cc:359] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42d47f6474a44fcbcb60900eec90d2e" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38791 } }
I20260131 03:03:29.715668 16208 raft_consensus.cc:385] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:29.715720 16208 raft_consensus.cc:740] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42d47f6474a44fcbcb60900eec90d2e, State: Initialized, Role: FOLLOWER
I20260131 03:03:29.716195 16208 consensus_queue.cc:260] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42d47f6474a44fcbcb60900eec90d2e" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38791 } }
I20260131 03:03:29.716352 16208 raft_consensus.cc:399] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:03:29.716408 16208 raft_consensus.cc:493] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:03:29.716487 16208 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:29.717633 16208 raft_consensus.cc:515] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42d47f6474a44fcbcb60900eec90d2e" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38791 } }
I20260131 03:03:29.717996 16208 leader_election.cc:304] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a42d47f6474a44fcbcb60900eec90d2e; no voters:
I20260131 03:03:29.718327 16208 leader_election.cc:290] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:03:29.718493 16213 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:03:29.718780 16213 raft_consensus.cc:697] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [term 1 LEADER]: Becoming Leader. State: Replica: a42d47f6474a44fcbcb60900eec90d2e, State: Running, Role: LEADER
I20260131 03:03:29.719117 16213 consensus_queue.cc:237] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42d47f6474a44fcbcb60900eec90d2e" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38791 } }
I20260131 03:03:29.719640 16208 sys_catalog.cc:565] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:03:29.720835 16214 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a42d47f6474a44fcbcb60900eec90d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42d47f6474a44fcbcb60900eec90d2e" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38791 } } }
I20260131 03:03:29.720966 16214 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:29.721401 16222 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:03:29.722453 16215 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a42d47f6474a44fcbcb60900eec90d2e. Latest consensus state: current_term: 1 leader_uuid: "a42d47f6474a44fcbcb60900eec90d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42d47f6474a44fcbcb60900eec90d2e" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 38791 } } }
I20260131 03:03:29.722559 16215 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:29.724221 16222 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:03:29.729806 16222 catalog_manager.cc:1357] Generated new cluster ID: 50f9624f945e400b87383327bbf291db
I20260131 03:03:29.729905 16222 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:03:29.743250 16222 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:03:29.744295 16222 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:03:29.753834 16222 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a42d47f6474a44fcbcb60900eec90d2e: Generated new TSK 0
I20260131 03:03:29.754700 16222 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:03:29.762629 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:38791
--builtin_ntp_servers=127.6.22.84:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260131 03:03:29.879153 16232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:29.879420 16232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:29.879473 16232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:29.883314 16232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:29.883414 16232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:29.883527 16232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:03:29.887791 16232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:38791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:29.889149 16232 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:29.890386 16232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:29.897540 16238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:29.897575 16240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:29.897877 16232 server_base.cc:1060] running on GCE node
W20260131 03:03:29.897534 16237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:29.898343 16232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:29.898977 16232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:29.900175 16232 hybrid_clock.cc:648] HybridClock initialized: now 1769828609900161 us; error 38 us; skew 500 ppm
I20260131 03:03:29.902369 16232 webserver.cc:492] Webserver started at http://127.6.22.65:34207/ using document root <none> and password file <none>
I20260131 03:03:29.902993 16232 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:29.903084 16232 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:29.903314 16232 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:29.905180 16232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "ef66862162f94ca7b9c5ebb74975c636"
format_stamp: "Formatted at 2026-01-31 03:03:29 on dist-test-slave-j368"
server_key: "7c4ce7802243108a7ec7a5d08666b255"
server_key_iv: "3e509a3d05062e55e2382980f455da8c"
server_key_version: "encryptionkey@0"
I20260131 03:03:29.905704 16232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "ef66862162f94ca7b9c5ebb74975c636"
format_stamp: "Formatted at 2026-01-31 03:03:29 on dist-test-slave-j368"
server_key: "7c4ce7802243108a7ec7a5d08666b255"
server_key_iv: "3e509a3d05062e55e2382980f455da8c"
server_key_version: "encryptionkey@0"
I20260131 03:03:29.910045 16232 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260131 03:03:29.913350 16246 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:29.914659 16232 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260131 03:03:29.914821 16232 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "ef66862162f94ca7b9c5ebb74975c636"
format_stamp: "Formatted at 2026-01-31 03:03:29 on dist-test-slave-j368"
server_key: "7c4ce7802243108a7ec7a5d08666b255"
server_key_iv: "3e509a3d05062e55e2382980f455da8c"
server_key_version: "encryptionkey@0"
I20260131 03:03:29.914948 16232 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:29.934003 16232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:29.934772 16232 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:29.934988 16232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:29.935580 16232 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:29.936707 16232 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:29.936791 16232 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:29.936892 16232 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:29.936941 16232 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:29.947983 16232 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:33249
I20260131 03:03:29.948022 16359 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:33249 every 8 connection(s)
I20260131 03:03:29.949519 16232 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:03:29.959573 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 16232
I20260131 03:03:29.959735 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:03:29.959844 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:29.952100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:53491 (local address 127.6.22.126:38791)
0131 03:03:29.952291 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:29.952295 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:29.953560 (+ 1265us) server_negotiation.cc:408] Connection header received
0131 03:03:29.954518 (+ 958us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:29.954522 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:29.954594 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:29.954708 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:29.955790 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.956316 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:29.957093 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.957281 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:29.959096 (+ 1815us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:29.959134 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:29.959137 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:03:29.959160 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:29.959189 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:03:29.959199 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:29.959293 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:29.959523 (+ 230us) server_negotiation.cc:300] Negotiation successful
0131 03:03:29.959644 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260131 03:03:29.960072 6233 external_mini_cluster.cc:1468] Setting key 5666cdaa08693aa054ed8ffaac4c987f
I20260131 03:03:29.960844 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:29.952386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38791 (local address 127.6.22.65:53491)
0131 03:03:29.953391 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:29.953441 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:29.954288 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:29.954900 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:29.954909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:29.954995 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:29.955613 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:29.955627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.956469 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:29.956472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:29.956958 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:29.956967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:29.957249 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:29.958649 (+ 1400us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:03:29.958652 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:29.958711 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:03:29.958815 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:29.959312 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:29.959323 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:29.959332 (+ 9us) client_negotiation.cc:769] Sending connection context
0131 03:03:29.959534 (+ 202us) client_negotiation.cc:240] Negotiation successful
0131 03:03:29.959908 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":155,"threads_started":1}
I20260131 03:03:29.962538 16360 heartbeater.cc:344] Connected to a master server at 127.6.22.126:38791
I20260131 03:03:29.962574 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:38791
--builtin_ntp_servers=127.6.22.84:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260131 03:03:29.962942 16360 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:29.963799 16360 heartbeater.cc:507] Master 127.6.22.126:38791 requested a full tablet report, sending...
I20260131 03:03:29.966318 16172 ts_manager.cc:194] Registered new tserver with Master: ef66862162f94ca7b9c5ebb74975c636 (127.6.22.65:33249)
I20260131 03:03:29.968235 16172 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.22.65:53491
W20260131 03:03:30.086102 16363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:30.086429 16363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:30.086489 16363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:30.090261 16363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:30.090340 16363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:30.090420 16363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:03:30.094718 16363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:38791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:30.096279 16363 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:30.097771 16363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:30.106106 16368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:30.106127 16369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:30.106127 16371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:30.106683 16363 server_base.cc:1060] running on GCE node
I20260131 03:03:30.107237 16363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:30.107959 16363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:30.109148 16363 hybrid_clock.cc:648] HybridClock initialized: now 1769828610109131 us; error 51 us; skew 500 ppm
I20260131 03:03:30.111326 16363 webserver.cc:492] Webserver started at http://127.6.22.66:34625/ using document root <none> and password file <none>
I20260131 03:03:30.112154 16363 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:30.112233 16363 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:30.112478 16363 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:30.114459 16363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "427430675c8a4aa3a98d50d7c5cf3bf8"
format_stamp: "Formatted at 2026-01-31 03:03:30 on dist-test-slave-j368"
server_key: "05ade014b34c80a695114302df44d349"
server_key_iv: "1b8b94cccab984df3190a4825d9cd1a4"
server_key_version: "encryptionkey@0"
I20260131 03:03:30.115020 16363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "427430675c8a4aa3a98d50d7c5cf3bf8"
format_stamp: "Formatted at 2026-01-31 03:03:30 on dist-test-slave-j368"
server_key: "05ade014b34c80a695114302df44d349"
server_key_iv: "1b8b94cccab984df3190a4825d9cd1a4"
server_key_version: "encryptionkey@0"
I20260131 03:03:30.119086 16363 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260131 03:03:30.122210 16377 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:30.123375 16363 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:03:30.123533 16363 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "427430675c8a4aa3a98d50d7c5cf3bf8"
format_stamp: "Formatted at 2026-01-31 03:03:30 on dist-test-slave-j368"
server_key: "05ade014b34c80a695114302df44d349"
server_key_iv: "1b8b94cccab984df3190a4825d9cd1a4"
server_key_version: "encryptionkey@0"
I20260131 03:03:30.123656 16363 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:30.161640 16363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:30.162392 16363 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:30.162608 16363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:30.163231 16363 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:30.164366 16363 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:30.164446 16363 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:30.164521 16363 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:30.164568 16363 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:30.174903 16363 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:46813
I20260131 03:03:30.174923 16490 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:46813 every 8 connection(s)
I20260131 03:03:30.175973 16363 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:03:30.181562 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 16363
I20260131 03:03:30.181742 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:03:30.182053 6233 external_mini_cluster.cc:1468] Setting key 2f87ca3e9966aa8cbf3b6928f56ef963
I20260131 03:03:30.184844 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:38791
--builtin_ntp_servers=127.6.22.84:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260131 03:03:30.187191 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:30.178270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:42069 (local address 127.6.22.126:38791)
0131 03:03:30.178487 (+ 217us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:30.178492 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:30.179921 (+ 1429us) server_negotiation.cc:408] Connection header received
0131 03:03:30.180974 (+ 1053us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:30.180979 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:30.181044 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:30.181132 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:30.182509 (+ 1377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:30.183376 (+ 867us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:30.184468 (+ 1092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:30.184724 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:30.186428 (+ 1704us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:30.186453 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:30.186455 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:03:30.186474 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:30.186500 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:03:30.186510 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:30.186586 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:30.186866 (+ 280us) server_negotiation.cc:300] Negotiation successful
0131 03:03:30.186965 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260131 03:03:30.187954 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:30.178721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38791 (local address 127.6.22.66:42069)
0131 03:03:30.179700 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:30.179747 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:30.180721 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:30.181361 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:30.181372 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:30.181469 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:30.182343 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:30.182360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:30.183694 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:30.183700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:30.184308 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:30.184322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:30.184652 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:30.186005 (+ 1353us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:03:30.186030 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:30.186098 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:03:30.186213 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:30.186620 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:30.186632 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:30.186640 (+ 8us) client_negotiation.cc:769] Sending connection context
0131 03:03:30.186843 (+ 203us) client_negotiation.cc:240] Negotiation successful
0131 03:03:30.187148 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":198,"threads_started":1}
I20260131 03:03:30.189391 16491 heartbeater.cc:344] Connected to a master server at 127.6.22.126:38791
I20260131 03:03:30.189692 16491 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:30.190388 16491 heartbeater.cc:507] Master 127.6.22.126:38791 requested a full tablet report, sending...
I20260131 03:03:30.191525 16172 ts_manager.cc:194] Registered new tserver with Master: 427430675c8a4aa3a98d50d7c5cf3bf8 (127.6.22.66:46813)
I20260131 03:03:30.192152 16172 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.22.66:42069
W20260131 03:03:30.308588 16494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:30.308964 16494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:30.309018 16494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:30.312542 16494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:30.312623 16494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:30.312796 16494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:03:30.317081 16494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:38791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:30.318393 16494 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:30.320518 16494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:30.330272 16499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:30.330456 16494 server_base.cc:1060] running on GCE node
W20260131 03:03:30.330274 16500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:30.330274 16502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:30.331128 16494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:30.331799 16494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:30.333009 16494 hybrid_clock.cc:648] HybridClock initialized: now 1769828610332995 us; error 35 us; skew 500 ppm
I20260131 03:03:30.335506 16494 webserver.cc:492] Webserver started at http://127.6.22.67:34143/ using document root <none> and password file <none>
I20260131 03:03:30.336182 16494 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:30.336248 16494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:30.336470 16494 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:30.338435 16494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "b7e4eba37e7b487fb6d068ff4ed21830"
format_stamp: "Formatted at 2026-01-31 03:03:30 on dist-test-slave-j368"
server_key: "698ec1e3a4cc15fc742dc55e97af5dfb"
server_key_iv: "f5fc6422dd60d38925d867960b142df0"
server_key_version: "encryptionkey@0"
I20260131 03:03:30.339020 16494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "b7e4eba37e7b487fb6d068ff4ed21830"
format_stamp: "Formatted at 2026-01-31 03:03:30 on dist-test-slave-j368"
server_key: "698ec1e3a4cc15fc742dc55e97af5dfb"
server_key_iv: "f5fc6422dd60d38925d867960b142df0"
server_key_version: "encryptionkey@0"
I20260131 03:03:30.343684 16494 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260131 03:03:30.346895 16508 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:30.348212 16494 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260131 03:03:30.348382 16494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "b7e4eba37e7b487fb6d068ff4ed21830"
format_stamp: "Formatted at 2026-01-31 03:03:30 on dist-test-slave-j368"
server_key: "698ec1e3a4cc15fc742dc55e97af5dfb"
server_key_iv: "f5fc6422dd60d38925d867960b142df0"
server_key_version: "encryptionkey@0"
I20260131 03:03:30.348532 16494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:30.375012 16494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:30.375797 16494 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:30.376029 16494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:30.376641 16494 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:30.377851 16494 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:30.377930 16494 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:30.378005 16494 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:30.378057 16494 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:30.389936 16494 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:39447
I20260131 03:03:30.389961 16621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:39447 every 8 connection(s)
I20260131 03:03:30.391150 16494 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:03:30.393399 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 16494
I20260131 03:03:30.393563 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:03:30.393870 6233 external_mini_cluster.cc:1468] Setting key 43a4ebc98ee63fd65e07ef74bd8577d1
I20260131 03:03:30.400518 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:30.393520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:49989 (local address 127.6.22.126:38791)
0131 03:03:30.393686 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:30.393690 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:30.394908 (+ 1218us) server_negotiation.cc:408] Connection header received
0131 03:03:30.395851 (+ 943us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:30.395855 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:30.395912 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:30.395979 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:30.397113 (+ 1134us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:30.397626 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:30.398299 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:30.398476 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:30.399935 (+ 1459us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:30.399966 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:30.399969 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:03:30.399984 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:30.400005 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:03:30.400013 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:30.400105 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:30.400288 (+ 183us) server_negotiation.cc:300] Negotiation successful
0131 03:03:30.400360 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260131 03:03:30.401288 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:30.393864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:38791 (local address 127.6.22.67:49989)
0131 03:03:30.394752 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:30.394811 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:30.395628 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:30.396152 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:30.396162 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:30.396249 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:30.396960 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:30.396975 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:30.397777 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:30.397780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:30.398176 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:30.398183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:30.398454 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:30.399657 (+ 1203us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:03:30.399659 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:30.399698 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:03:30.399765 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:30.400111 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:30.400121 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:30.400126 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:03:30.400298 (+ 172us) client_negotiation.cc:240] Negotiation successful
0131 03:03:30.400563 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":210,"threads_started":1}
I20260131 03:03:30.402479 16622 heartbeater.cc:344] Connected to a master server at 127.6.22.126:38791
I20260131 03:03:30.402799 16622 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:30.403504 16622 heartbeater.cc:507] Master 127.6.22.126:38791 requested a full tablet report, sending...
I20260131 03:03:30.404903 16172 ts_manager.cc:194] Registered new tserver with Master: b7e4eba37e7b487fb6d068ff4ed21830 (127.6.22.67:39447)
I20260131 03:03:30.405520 16172 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.22.67:49989
I20260131 03:03:30.408721 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260131 03:03:30.411818 16211 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:30.410873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.6.22.126:38791)
0131 03:03:30.411028 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:30.411031 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:30.411206 (+ 175us) server_negotiation.cc:408] Connection header received
0131 03:03:30.411315 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:30.411317 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:30.411364 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:30.411440 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:30.411641 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55870: BlockingRecv error: recv got EOF from 127.0.0.1:55870 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260131 03:03:30.412055 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 16232
I20260131 03:03:30.418524 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 16363
I20260131 03:03:30.424823 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 16494
I20260131 03:03:30.432070 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 16141
2026-01-31T03:03:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (945 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-01-31T03:03:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:03:30Z Disabled control of system clock
I20260131 03:03:30.465559 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:35465
--webserver_interface=127.6.22.126
--webserver_port=0
--builtin_ntp_servers=127.6.22.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:35465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260131 03:03:30.582312 16633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:30.582577 16633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:30.582628 16633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:30.586226 16633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:03:30.586318 16633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:30.586335 16633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:30.586355 16633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:03:30.586372 16633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:03:30.590655 16633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:35465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:30.591967 16633 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:30.593238 16633 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:30.600013 16641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:30.600121 16639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:30.600023 16638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:30.600360 16633 server_base.cc:1060] running on GCE node
I20260131 03:03:30.601244 16633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:30.602520 16633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:30.603734 16633 hybrid_clock.cc:648] HybridClock initialized: now 1769828610603691 us; error 62 us; skew 500 ppm
I20260131 03:03:30.606298 16633 webserver.cc:492] Webserver started at http://127.6.22.126:42297/ using document root <none> and password file <none>
I20260131 03:03:30.606976 16633 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:30.607048 16633 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:30.607270 16633 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:30.609241 16633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "bd7dbc29e06a4ee280623bd049290ba1"
format_stamp: "Formatted at 2026-01-31 03:03:30 on dist-test-slave-j368"
server_key: "9b72d95661b4c54d4e3c25170b4b9158"
server_key_iv: "9fd4945ea6e2fcef6f3926f02efe08f8"
server_key_version: "encryptionkey@0"
I20260131 03:03:30.609804 16633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "bd7dbc29e06a4ee280623bd049290ba1"
format_stamp: "Formatted at 2026-01-31 03:03:30 on dist-test-slave-j368"
server_key: "9b72d95661b4c54d4e3c25170b4b9158"
server_key_iv: "9fd4945ea6e2fcef6f3926f02efe08f8"
server_key_version: "encryptionkey@0"
I20260131 03:03:30.613648 16633 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260131 03:03:30.616415 16647 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:30.617882 16633 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:03:30.618075 16633 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "bd7dbc29e06a4ee280623bd049290ba1"
format_stamp: "Formatted at 2026-01-31 03:03:30 on dist-test-slave-j368"
server_key: "9b72d95661b4c54d4e3c25170b4b9158"
server_key_iv: "9fd4945ea6e2fcef6f3926f02efe08f8"
server_key_version: "encryptionkey@0"
I20260131 03:03:30.618199 16633 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:30.624078 16633 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:30.625098 16633 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:30.625326 16633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:30.634191 16633 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:35465
I20260131 03:03:30.634192 16699 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:35465 every 8 connection(s)
I20260131 03:03:30.635399 16633 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:03:30.638617 16700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:30.642345 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 16633
I20260131 03:03:30.642494 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:03:30.642812 6233 external_mini_cluster.cc:1468] Setting key b158f37c4b9eef6764160f3d2161bb72
I20260131 03:03:30.645351 16700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1: Bootstrap starting.
I20260131 03:03:30.647818 16700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:30.648587 16700 log.cc:826] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1: Log is configured to *not* fsync() on all Append() calls
I20260131 03:03:30.650895 16700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1: No bootstrap required, opened a new log
I20260131 03:03:30.652259 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:30.644385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.6.22.126:35465)
0131 03:03:30.645116 (+ 731us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:30.645126 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:30.645163 (+ 37us) server_negotiation.cc:408] Connection header received
0131 03:03:30.645757 (+ 594us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:30.645777 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:30.646109 (+ 332us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:30.646447 (+ 338us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:30.651044 (+ 4597us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:30.651051 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:30.651066 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:03:30.651082 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:30.651158 (+ 76us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:03:30.651173 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:30.651315 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:30.651419 (+ 104us) server_negotiation.cc:300] Negotiation successful
0131 03:03:30.651664 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":135,"threads_started":1}
I20260131 03:03:30.654006 16700 raft_consensus.cc:359] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7dbc29e06a4ee280623bd049290ba1" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 35465 } }
I20260131 03:03:30.654224 16700 raft_consensus.cc:385] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:30.654301 16700 raft_consensus.cc:740] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd7dbc29e06a4ee280623bd049290ba1, State: Initialized, Role: FOLLOWER
I20260131 03:03:30.654795 16700 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7dbc29e06a4ee280623bd049290ba1" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 35465 } }
I20260131 03:03:30.654994 16700 raft_consensus.cc:399] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:03:30.655062 16700 raft_consensus.cc:493] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:03:30.655154 16700 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:30.656090 16700 raft_consensus.cc:515] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7dbc29e06a4ee280623bd049290ba1" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 35465 } }
I20260131 03:03:30.656461 16700 leader_election.cc:304] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd7dbc29e06a4ee280623bd049290ba1; no voters:
I20260131 03:03:30.656840 16700 leader_election.cc:290] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:03:30.656962 16705 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:03:30.657204 16705 raft_consensus.cc:697] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [term 1 LEADER]: Becoming Leader. State: Replica: bd7dbc29e06a4ee280623bd049290ba1, State: Running, Role: LEADER
I20260131 03:03:30.657552 16705 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7dbc29e06a4ee280623bd049290ba1" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 35465 } }
I20260131 03:03:30.658002 16700 sys_catalog.cc:565] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:03:30.659209 16707 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd7dbc29e06a4ee280623bd049290ba1. Latest consensus state: current_term: 1 leader_uuid: "bd7dbc29e06a4ee280623bd049290ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7dbc29e06a4ee280623bd049290ba1" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 35465 } } }
I20260131 03:03:30.659391 16707 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:30.659232 16706 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd7dbc29e06a4ee280623bd049290ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7dbc29e06a4ee280623bd049290ba1" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 35465 } } }
I20260131 03:03:30.659520 16706 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1 [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:30.660279 16714 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:03:30.663386 16714 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:03:30.670199 16714 catalog_manager.cc:1357] Generated new cluster ID: adc7d50080a94126b1d2712fc02eb2f0
I20260131 03:03:30.670337 16714 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:03:30.683404 16714 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:03:30.683825 16714 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:03:30.691004 16714 catalog_manager.cc:6027] T 00000000000000000000000000000000 P bd7dbc29e06a4ee280623bd049290ba1: Generated new TSK 0
I20260131 03:03:30.691980 16714 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:03:30.703094 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:35465
--builtin_ntp_servers=127.6.22.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260131 03:03:30.821086 16724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:30.821414 16724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:30.821509 16724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:30.825295 16724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:30.825398 16724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:30.825518 16724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:03:30.829694 16724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:35465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:30.831125 16724 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:30.832525 16724 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:30.840528 16729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:30.840538 16732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:30.840736 16724 server_base.cc:1060] running on GCE node
W20260131 03:03:30.840540 16730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:30.841317 16724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:30.842041 16724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:30.843241 16724 hybrid_clock.cc:648] HybridClock initialized: now 1769828610843207 us; error 83 us; skew 500 ppm
I20260131 03:03:30.845511 16724 webserver.cc:492] Webserver started at http://127.6.22.65:37671/ using document root <none> and password file <none>
I20260131 03:03:30.846230 16724 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:30.846292 16724 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:30.846531 16724 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:30.848608 16724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "9d85e4ae8a004b9f90710a93a812acbb"
format_stamp: "Formatted at 2026-01-31 03:03:30 on dist-test-slave-j368"
server_key: "e58a7dc56dc206b0ae6064726bfef31e"
server_key_iv: "f92c90d83ccb397b9db27eab70b361f1"
server_key_version: "encryptionkey@0"
I20260131 03:03:30.849304 16724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "9d85e4ae8a004b9f90710a93a812acbb"
format_stamp: "Formatted at 2026-01-31 03:03:30 on dist-test-slave-j368"
server_key: "e58a7dc56dc206b0ae6064726bfef31e"
server_key_iv: "f92c90d83ccb397b9db27eab70b361f1"
server_key_version: "encryptionkey@0"
I20260131 03:03:30.853785 16724 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260131 03:03:30.856860 16738 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:30.858044 16724 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260131 03:03:30.858201 16724 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "9d85e4ae8a004b9f90710a93a812acbb"
format_stamp: "Formatted at 2026-01-31 03:03:30 on dist-test-slave-j368"
server_key: "e58a7dc56dc206b0ae6064726bfef31e"
server_key_iv: "f92c90d83ccb397b9db27eab70b361f1"
server_key_version: "encryptionkey@0"
I20260131 03:03:30.858333 16724 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:30.865206 16724 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:30.866004 16724 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:30.866228 16724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:30.866891 16724 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:30.868106 16724 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:30.868156 16724 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:30.868201 16724 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:30.868258 16724 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:30.880097 16724 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:34603
I20260131 03:03:30.880178 16851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:34603 every 8 connection(s)
I20260131 03:03:30.881575 16724 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:03:30.888561 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:30.883964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:60855 (local address 127.6.22.126:35465)
0131 03:03:30.884145 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:30.884150 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:30.885552 (+ 1402us) server_negotiation.cc:408] Connection header received
0131 03:03:30.886449 (+ 897us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:30.886453 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:30.886522 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:30.886638 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:30.887958 (+ 1320us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:30.887962 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:30.887965 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:03:30.887979 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:30.888005 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:03:30.888014 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:30.888121 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:30.888325 (+ 204us) server_negotiation.cc:300] Negotiation successful
0131 03:03:30.888409 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260131 03:03:30.889425 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:30.884240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:35465 (local address 127.6.22.65:60855)
0131 03:03:30.885315 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:30.885367 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:30.886190 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:30.886826 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:30.886835 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:30.886936 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:30.887620 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:03:30.887622 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:30.887660 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:03:30.887726 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:30.888121 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:30.888129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:30.888133 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:03:30.888339 (+ 206us) client_negotiation.cc:240] Negotiation successful
0131 03:03:30.888555 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":121,"threads_started":1}
I20260131 03:03:30.890133 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 16724
I20260131 03:03:30.890278 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:03:30.890548 6233 external_mini_cluster.cc:1468] Setting key cfa057ef47e82c9a844a4e5841d4d934
I20260131 03:03:30.891027 16852 heartbeater.cc:344] Connected to a master server at 127.6.22.126:35465
I20260131 03:03:30.891340 16852 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:30.891889 16852 heartbeater.cc:507] Master 127.6.22.126:35465 requested a full tablet report, sending...
I20260131 03:03:30.892789 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:35465
--builtin_ntp_servers=127.6.22.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260131 03:03:30.893800 16664 ts_manager.cc:194] Registered new tserver with Master: 9d85e4ae8a004b9f90710a93a812acbb (127.6.22.65:34603)
W20260131 03:03:31.010129 16855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:31.010391 16855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:31.010488 16855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:31.014060 16855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:31.014144 16855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:31.014266 16855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:03:31.018507 16855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:35465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:31.019847 16855 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:31.021200 16855 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:31.029274 16860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:31.029273 16861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:31.029573 16855 server_base.cc:1060] running on GCE node
W20260131 03:03:31.029281 16863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:31.030056 16855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:31.030689 16855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:31.031868 16855 hybrid_clock.cc:648] HybridClock initialized: now 1769828611031842 us; error 51 us; skew 500 ppm
I20260131 03:03:31.034117 16855 webserver.cc:492] Webserver started at http://127.6.22.66:45519/ using document root <none> and password file <none>
I20260131 03:03:31.034771 16855 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:31.034832 16855 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:31.035068 16855 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:31.037041 16855 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "5c82d50a46cd4854afd810dad92e8ccc"
format_stamp: "Formatted at 2026-01-31 03:03:31 on dist-test-slave-j368"
server_key: "e5239db4a9faf7027f0d7f28428b3f01"
server_key_iv: "06d62e03f17ca29d9d127710d184c33e"
server_key_version: "encryptionkey@0"
I20260131 03:03:31.037595 16855 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "5c82d50a46cd4854afd810dad92e8ccc"
format_stamp: "Formatted at 2026-01-31 03:03:31 on dist-test-slave-j368"
server_key: "e5239db4a9faf7027f0d7f28428b3f01"
server_key_iv: "06d62e03f17ca29d9d127710d184c33e"
server_key_version: "encryptionkey@0"
I20260131 03:03:31.041702 16855 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260131 03:03:31.044198 16869 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:31.045360 16855 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:03:31.045504 16855 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "5c82d50a46cd4854afd810dad92e8ccc"
format_stamp: "Formatted at 2026-01-31 03:03:31 on dist-test-slave-j368"
server_key: "e5239db4a9faf7027f0d7f28428b3f01"
server_key_iv: "06d62e03f17ca29d9d127710d184c33e"
server_key_version: "encryptionkey@0"
I20260131 03:03:31.045621 16855 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:31.050881 16855 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:31.051573 16855 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:31.051772 16855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:31.052363 16855 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:31.053493 16855 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:31.053544 16855 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:31.053615 16855 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:31.053651 16855 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:31.064150 16855 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:36123
I20260131 03:03:31.064172 16982 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:36123 every 8 connection(s)
I20260131 03:03:31.065413 16855 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:03:31.069936 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 16855
I20260131 03:03:31.070173 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:03:31.070500 6233 external_mini_cluster.cc:1468] Setting key cf09b79e83d0dd285527550268a1152b
I20260131 03:03:31.071907 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:31.067383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:33859 (local address 127.6.22.126:35465)
0131 03:03:31.067581 (+ 198us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:31.067587 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:31.068602 (+ 1015us) server_negotiation.cc:408] Connection header received
0131 03:03:31.069502 (+ 900us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:31.069506 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:31.069557 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:31.069675 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:31.070939 (+ 1264us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:31.070942 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:31.070944 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:03:31.070958 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:31.070984 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:03:31.070992 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:31.071122 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:31.071733 (+ 611us) server_negotiation.cc:300] Negotiation successful
0131 03:03:31.071778 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260131 03:03:31.072468 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:31.067731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:35465 (local address 127.6.22.66:33859)
0131 03:03:31.068441 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:31.068480 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:31.069286 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:31.069808 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:31.069817 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:31.069907 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:31.070627 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:03:31.070629 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:31.070687 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:03:31.070752 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:31.071121 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:31.071129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:31.071133 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:03:31.071344 (+ 211us) client_negotiation.cc:240] Negotiation successful
0131 03:03:31.071548 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260131 03:03:31.072798 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:35465
--builtin_ntp_servers=127.6.22.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260131 03:03:31.073737 16983 heartbeater.cc:344] Connected to a master server at 127.6.22.126:35465
I20260131 03:03:31.074033 16983 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:31.074492 16983 heartbeater.cc:507] Master 127.6.22.126:35465 requested a full tablet report, sending...
I20260131 03:03:31.075659 16664 ts_manager.cc:194] Registered new tserver with Master: 5c82d50a46cd4854afd810dad92e8ccc (127.6.22.66:36123)
W20260131 03:03:31.189471 16986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:31.189725 16986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:31.189783 16986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:31.193605 16986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:31.193683 16986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:31.193765 16986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:03:31.197868 16986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:35465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:31.199014 16986 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:31.200183 16986 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:31.208837 16991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:31.208839 16992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:31.208837 16994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:31.209559 16986 server_base.cc:1060] running on GCE node
I20260131 03:03:31.210024 16986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:31.210664 16986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:31.211864 16986 hybrid_clock.cc:648] HybridClock initialized: now 1769828611211837 us; error 52 us; skew 500 ppm
I20260131 03:03:31.214231 16986 webserver.cc:492] Webserver started at http://127.6.22.67:39795/ using document root <none> and password file <none>
I20260131 03:03:31.214869 16986 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:31.214962 16986 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:31.215189 16986 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:31.217080 16986 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "1b613dbb319d4514944b679043149324"
format_stamp: "Formatted at 2026-01-31 03:03:31 on dist-test-slave-j368"
server_key: "cec78b0d5a8b545ecc5fcdca73ae80b4"
server_key_iv: "7beea6b862be437b06af0e0d233506b4"
server_key_version: "encryptionkey@0"
I20260131 03:03:31.217587 16986 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "1b613dbb319d4514944b679043149324"
format_stamp: "Formatted at 2026-01-31 03:03:31 on dist-test-slave-j368"
server_key: "cec78b0d5a8b545ecc5fcdca73ae80b4"
server_key_iv: "7beea6b862be437b06af0e0d233506b4"
server_key_version: "encryptionkey@0"
I20260131 03:03:31.223313 16986 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260131 03:03:31.226570 17000 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:31.228327 16986 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260131 03:03:31.228536 16986 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "1b613dbb319d4514944b679043149324"
format_stamp: "Formatted at 2026-01-31 03:03:31 on dist-test-slave-j368"
server_key: "cec78b0d5a8b545ecc5fcdca73ae80b4"
server_key_iv: "7beea6b862be437b06af0e0d233506b4"
server_key_version: "encryptionkey@0"
I20260131 03:03:31.228747 16986 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:31.234603 16986 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:31.235527 16986 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:31.235812 16986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:31.236439 16986 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:31.237567 16986 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:31.237641 16986 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:31.237704 16986 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:31.237742 16986 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:31.248524 16986 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:38553
I20260131 03:03:31.248643 17113 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:38553 every 8 connection(s)
I20260131 03:03:31.249915 16986 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:03:31.250557 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 16986
I20260131 03:03:31.250676 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:03:31.251006 6233 external_mini_cluster.cc:1468] Setting key e4eda12770a17e74e675e7e05984aa9e
I20260131 03:03:31.255823 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:31.251989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:50927 (local address 127.6.22.126:35465)
0131 03:03:31.252153 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:31.252158 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:31.253358 (+ 1200us) server_negotiation.cc:408] Connection header received
0131 03:03:31.254175 (+ 817us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:31.254178 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:31.254217 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:31.254294 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:31.255383 (+ 1089us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:31.255386 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:31.255388 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:03:31.255399 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:31.255425 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:03:31.255432 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:31.255532 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:31.255668 (+ 136us) server_negotiation.cc:300] Negotiation successful
0131 03:03:31.255708 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260131 03:03:31.256855 17116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:31.252407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:35465 (local address 127.6.22.67:50927)
0131 03:03:31.253202 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:31.253257 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:31.253949 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:31.254443 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:31.254450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:31.254532 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:31.255140 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:03:31.255142 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:31.255175 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:03:31.255229 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:31.255536 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:31.255542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:31.255545 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:03:31.255713 (+ 168us) client_negotiation.cc:240] Negotiation successful
0131 03:03:31.256038 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":154,"threads_started":1}
I20260131 03:03:31.258188 17114 heartbeater.cc:344] Connected to a master server at 127.6.22.126:35465
I20260131 03:03:31.258477 17114 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:31.258963 17114 heartbeater.cc:507] Master 127.6.22.126:35465 requested a full tablet report, sending...
I20260131 03:03:31.260093 16664 ts_manager.cc:194] Registered new tserver with Master: 1b613dbb319d4514944b679043149324 (127.6.22.67:38553)
I20260131 03:03:31.266140 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260131 03:03:31.269871 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:03:31.268619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.6.22.126:35465)
0131 03:03:31.268874 (+ 255us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:31.268878 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:31.269098 (+ 220us) server_negotiation.cc:408] Connection header received
0131 03:03:31.269172 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:31.269175 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:31.269241 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:31.269335 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:31.269711 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52536: BlockingRecv error: recv got EOF from 127.0.0.1:52536 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
I20260131 03:03:31.270097 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 16724
I20260131 03:03:31.275970 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 16855
I20260131 03:03:31.281607 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 16986
I20260131 03:03:31.287361 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 16633
2026-01-31T03:03:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (853 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:03:31 dist-test-slave-j368 krb5kdc[17124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:03:31 dist-test-slave-j368 krb5kdc[17124](info): set up 2 sockets
Jan 31 03:03:31 dist-test-slave-j368 krb5kdc[17124](info): commencing operation
krb5kdc: starting...
W20260131 03:03:33.303442 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.993s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:03:33 dist-test-slave-j368 krb5kdc[17124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:03:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:03:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:03:33.450196 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:40445
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:38189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:40445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:33.561564 17140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:33.561911 17140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:33.562018 17140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:33.565675 17140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:03:33.565804 17140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:33.565850 17140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:33.565891 17140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:03:33.565933 17140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:03:33.570593 17140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:38189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:40445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:33.571781 17140 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:33.572717 17140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:33.578588 17145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:33.578604 17146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:33.578596 17148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:33.578728 17140 server_base.cc:1060] running on GCE node
I20260131 03:03:33.579541 17140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:33.580549 17140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:33.581740 17140 hybrid_clock.cc:648] HybridClock initialized: now 1769828613581718 us; error 45 us; skew 500 ppm
Jan 31 03:03:33 dist-test-slave-j368 krb5kdc[17124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828613, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:33.584432 17140 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:03:33.585661 17140 webserver.cc:492] Webserver started at http://127.6.22.126:38781/ using document root <none> and password file <none>
I20260131 03:03:33.586207 17140 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:33.586254 17140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:33.586418 17140 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:33.588141 17140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "e118088fc49e4ccd9c4837034aeaf217"
format_stamp: "Formatted at 2026-01-31 03:03:33 on dist-test-slave-j368"
server_key: "2e7e416fc4a4a6f3882f4e5b4a9a94fa"
server_key_iv: "1156303eb2131f9d1f0c79fe0eff9fda"
server_key_version: "encryptionkey@0"
I20260131 03:03:33.588604 17140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "e118088fc49e4ccd9c4837034aeaf217"
format_stamp: "Formatted at 2026-01-31 03:03:33 on dist-test-slave-j368"
server_key: "2e7e416fc4a4a6f3882f4e5b4a9a94fa"
server_key_iv: "1156303eb2131f9d1f0c79fe0eff9fda"
server_key_version: "encryptionkey@0"
I20260131 03:03:33.592397 17140 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:03:33.595201 17155 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:33.596462 17140 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260131 03:03:33.596632 17140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "e118088fc49e4ccd9c4837034aeaf217"
format_stamp: "Formatted at 2026-01-31 03:03:33 on dist-test-slave-j368"
server_key: "2e7e416fc4a4a6f3882f4e5b4a9a94fa"
server_key_iv: "1156303eb2131f9d1f0c79fe0eff9fda"
server_key_version: "encryptionkey@0"
I20260131 03:03:33.596822 17140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:33.618642 17140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:33.621803 17140 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:33.622038 17140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:33.630545 17140 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:40445
I20260131 03:03:33.630545 17207 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:40445 every 8 connection(s)
I20260131 03:03:33.631721 17140 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:03:33.634724 17208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:33.637542 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 17140
I20260131 03:03:33.637689 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:03:33.637987 6233 external_mini_cluster.cc:1468] Setting key 04546b45ee8e8cd9a205647160b0bed0
I20260131 03:03:33.641834 17208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217: Bootstrap starting.
I20260131 03:03:33.644536 17208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:33.645608 17208 log.cc:826] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217: Log is configured to *not* fsync() on all Append() calls
Jan 31 03:03:33 dist-test-slave-j368 krb5kdc[17124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:03:33.648085 17208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217: No bootstrap required, opened a new log
I20260131 03:03:33.651639 17208 raft_consensus.cc:359] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e118088fc49e4ccd9c4837034aeaf217" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 40445 } }
I20260131 03:03:33.651881 17208 raft_consensus.cc:385] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:33.651942 17208 raft_consensus.cc:740] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e118088fc49e4ccd9c4837034aeaf217, State: Initialized, Role: FOLLOWER
I20260131 03:03:33.652037 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:33.639426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.6.22.126:40445)
0131 03:03:33.639873 (+ 447us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:33.639884 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:33.639922 (+ 38us) server_negotiation.cc:408] Connection header received
0131 03:03:33.640613 (+ 691us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:33.640634 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:33.641028 (+ 394us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:33.641327 (+ 299us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:33.642426 (+ 1099us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:33.643166 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:33.644086 (+ 920us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:33.644308 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:33.647070 (+ 2762us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:33.647095 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:33.647108 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:33.647159 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:33.649251 (+ 2092us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:33.649696 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:33.649704 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:33.649712 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:33.649815 (+ 103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:33.650039 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:33.650042 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:33.650044 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:33.650504 (+ 460us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:33.650725 (+ 221us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:33.650939 (+ 214us) server_negotiation.cc:300] Negotiation successful
0131 03:03:33.651227 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":152,"threads_started":1}
I20260131 03:03:33.652456 17208 consensus_queue.cc:260] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e118088fc49e4ccd9c4837034aeaf217" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 40445 } }
I20260131 03:03:33.652617 17208 raft_consensus.cc:399] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:03:33.652720 17208 raft_consensus.cc:493] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:03:33.652814 17208 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:33.653965 17208 raft_consensus.cc:515] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e118088fc49e4ccd9c4837034aeaf217" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 40445 } }
I20260131 03:03:33.654366 17208 leader_election.cc:304] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e118088fc49e4ccd9c4837034aeaf217; no voters:
I20260131 03:03:33.654708 17208 leader_election.cc:290] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:03:33.654837 17213 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:03:33.655071 17213 raft_consensus.cc:697] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [term 1 LEADER]: Becoming Leader. State: Replica: e118088fc49e4ccd9c4837034aeaf217, State: Running, Role: LEADER
I20260131 03:03:33.655373 17213 consensus_queue.cc:237] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e118088fc49e4ccd9c4837034aeaf217" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 40445 } }
I20260131 03:03:33.655920 17208 sys_catalog.cc:565] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:03:33.657217 17214 sys_catalog.cc:455] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e118088fc49e4ccd9c4837034aeaf217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e118088fc49e4ccd9c4837034aeaf217" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 40445 } } }
I20260131 03:03:33.657284 17215 sys_catalog.cc:455] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e118088fc49e4ccd9c4837034aeaf217. Latest consensus state: current_term: 1 leader_uuid: "e118088fc49e4ccd9c4837034aeaf217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e118088fc49e4ccd9c4837034aeaf217" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 40445 } } }
I20260131 03:03:33.657358 17214 sys_catalog.cc:458] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:33.657377 17215 sys_catalog.cc:458] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217 [sys.catalog]: This master's current role is: LEADER
I20260131 03:03:33.657871 17222 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:03:33.661207 17222 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:03:33.667124 17222 catalog_manager.cc:1357] Generated new cluster ID: 2ebdae0b1eb14f06b0b529841756832d
I20260131 03:03:33.667235 17222 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:03:33.678609 17222 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:03:33.679629 17222 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:03:33.685758 17222 catalog_manager.cc:6027] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217: Generated new TSK 0
I20260131 03:03:33.686687 17222 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:03:33.744900 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:40445
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:38189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:33.863202 17236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:33.863673 17236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:33.863803 17236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:33.867617 17236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:33.867725 17236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:33.867816 17236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:03:33.872808 17236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:38189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:33.874440 17236 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:33.875598 17236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:33.883169 17241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:33.883175 17242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:33.883169 17244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:33.883486 17236 server_base.cc:1060] running on GCE node
I20260131 03:03:33.884053 17236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:33.884871 17236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:33.886106 17236 hybrid_clock.cc:648] HybridClock initialized: now 1769828613886093 us; error 41 us; skew 500 ppm
Jan 31 03:03:33 dist-test-slave-j368 krb5kdc[17124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828613, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:33.889325 17236 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:03:33.890611 17236 webserver.cc:492] Webserver started at http://127.6.22.65:42363/ using document root <none> and password file <none>
I20260131 03:03:33.891274 17236 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:33.891361 17236 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:33.891589 17236 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:33.893522 17236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "8478165796104a27a780d4d1eeef7c28"
format_stamp: "Formatted at 2026-01-31 03:03:33 on dist-test-slave-j368"
server_key: "f1b58af565bfdce6855357268b7dd213"
server_key_iv: "07a6b638c0978548ef298f8393ad7e5d"
server_key_version: "encryptionkey@0"
I20260131 03:03:33.894172 17236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "8478165796104a27a780d4d1eeef7c28"
format_stamp: "Formatted at 2026-01-31 03:03:33 on dist-test-slave-j368"
server_key: "f1b58af565bfdce6855357268b7dd213"
server_key_iv: "07a6b638c0978548ef298f8393ad7e5d"
server_key_version: "encryptionkey@0"
I20260131 03:03:33.898360 17236 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260131 03:03:33.901865 17251 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:33.903370 17236 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260131 03:03:33.903564 17236 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "8478165796104a27a780d4d1eeef7c28"
format_stamp: "Formatted at 2026-01-31 03:03:33 on dist-test-slave-j368"
server_key: "f1b58af565bfdce6855357268b7dd213"
server_key_iv: "07a6b638c0978548ef298f8393ad7e5d"
server_key_version: "encryptionkey@0"
I20260131 03:03:33.903719 17236 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:33.920219 17236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:33.923578 17236 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:33.923828 17236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:33.924453 17236 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:33.925627 17236 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:33.925702 17236 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:33.925769 17236 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:33.925819 17236 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:33.936986 17236 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:38501
I20260131 03:03:33.937346 17364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:38501 every 8 connection(s)
I20260131 03:03:33.938194 17236 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:03:33.942025 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 17236
I20260131 03:03:33.942195 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:03:33.942431 6233 external_mini_cluster.cc:1468] Setting key db9fa0df4f95f6ccaf797d0ca157f839
Jan 31 03:03:33 dist-test-slave-j368 krb5kdc[17124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828613, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
I20260131 03:03:33.951987 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:33.940199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:60189 (local address 127.6.22.126:40445)
0131 03:03:33.940359 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:33.940363 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:33.941083 (+ 720us) server_negotiation.cc:408] Connection header received
0131 03:03:33.942013 (+ 930us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:33.942016 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:33.942078 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:33.942184 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:33.944061 (+ 1877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:33.944585 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:33.945327 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:33.945515 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:33.948001 (+ 2486us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:33.948028 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:33.948030 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:33.948060 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:33.949706 (+ 1646us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:33.950306 (+ 600us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:33.950312 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:33.950314 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:33.950369 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:33.950713 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:33.950716 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:33.950718 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:33.950924 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:33.951038 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:33.951647 (+ 609us) server_negotiation.cc:300] Negotiation successful
0131 03:03:33.951803 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260131 03:03:33.952869 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:33.940417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:40445 (local address 127.6.22.65:60189)
0131 03:03:33.940924 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:33.940974 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:33.941784 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:33.942365 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:33.942373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:33.942854 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:33.943882 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:33.943899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:33.944790 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:33.944792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:33.945196 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:33.945204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:33.945487 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:33.946182 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:33.946212 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:33.947797 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:33.949877 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:33.949884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:33.949907 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:33.950188 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:33.950485 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:33.950488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:33.950489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:33.950601 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:33.951043 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:33.951053 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:33.951379 (+ 326us) client_negotiation.cc:769] Sending connection context
0131 03:03:33.951611 (+ 232us) client_negotiation.cc:240] Negotiation successful
0131 03:03:33.951911 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
I20260131 03:03:33.954227 17365 heartbeater.cc:344] Connected to a master server at 127.6.22.126:40445
I20260131 03:03:33.954540 17365 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:33.955284 17365 heartbeater.cc:507] Master 127.6.22.126:40445 requested a full tablet report, sending...
I20260131 03:03:33.957227 17170 ts_manager.cc:194] Registered new tserver with Master: 8478165796104a27a780d4d1eeef7c28 (127.6.22.65:38501)
I20260131 03:03:33.958593 17170 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:60189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:03:33.995150 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:40445
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:38189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:34.120882 17372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:34.121152 17372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:34.121224 17372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:34.124907 17372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:34.124991 17372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:34.125077 17372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:03:34.129706 17372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:38189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:34.130877 17372 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:34.131721 17372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:34.138651 17380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:34.138651 17377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:34.138652 17378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:34.139016 17372 server_base.cc:1060] running on GCE node
I20260131 03:03:34.139417 17372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:34.140050 17372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:34.141212 17372 hybrid_clock.cc:648] HybridClock initialized: now 1769828614141202 us; error 32 us; skew 500 ppm
Jan 31 03:03:34 dist-test-slave-j368 krb5kdc[17124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828614, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:34.144001 17372 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:03:34.145339 17372 webserver.cc:492] Webserver started at http://127.6.22.66:33003/ using document root <none> and password file <none>
I20260131 03:03:34.145946 17372 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:34.146000 17372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:34.146222 17372 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:34.148026 17372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "849ed8010cf5476b8823a224f9f54805"
format_stamp: "Formatted at 2026-01-31 03:03:34 on dist-test-slave-j368"
server_key: "83603f3bca0130e2bc6fda464942586f"
server_key_iv: "e158b63467a4cb732cd151f70711f589"
server_key_version: "encryptionkey@0"
I20260131 03:03:34.148583 17372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "849ed8010cf5476b8823a224f9f54805"
format_stamp: "Formatted at 2026-01-31 03:03:34 on dist-test-slave-j368"
server_key: "83603f3bca0130e2bc6fda464942586f"
server_key_iv: "e158b63467a4cb732cd151f70711f589"
server_key_version: "encryptionkey@0"
I20260131 03:03:34.152963 17372 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260131 03:03:34.155980 17387 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:34.157444 17372 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260131 03:03:34.157606 17372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "849ed8010cf5476b8823a224f9f54805"
format_stamp: "Formatted at 2026-01-31 03:03:34 on dist-test-slave-j368"
server_key: "83603f3bca0130e2bc6fda464942586f"
server_key_iv: "e158b63467a4cb732cd151f70711f589"
server_key_version: "encryptionkey@0"
I20260131 03:03:34.157748 17372 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:34.168911 17372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:34.171844 17372 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:34.172056 17372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:34.172698 17372 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:34.173645 17372 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:34.173693 17372 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:34.173761 17372 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:34.173794 17372 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:34.183873 17372 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:41451
I20260131 03:03:34.183903 17500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:41451 every 8 connection(s)
I20260131 03:03:34.185000 17372 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:03:34.192160 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 17372
I20260131 03:03:34.192320 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:03:34.192566 6233 external_mini_cluster.cc:1468] Setting key a94a1511e02b1ac89645f06c63687245
Jan 31 03:03:34 dist-test-slave-j368 krb5kdc[17124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828614, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:03:34.199877 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.187003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:50099 (local address 127.6.22.126:40445)
0131 03:03:34.187153 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:34.187157 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:34.187903 (+ 746us) server_negotiation.cc:408] Connection header received
0131 03:03:34.188959 (+ 1056us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:34.188964 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:34.189053 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:34.189228 (+ 175us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:34.191090 (+ 1862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.191641 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.192354 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.192552 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.195093 (+ 2541us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:34.195115 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:34.195117 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:34.195145 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:34.196855 (+ 1710us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:34.197987 (+ 1132us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:34.197991 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:34.197996 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:34.198056 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:34.198509 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:34.198511 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:34.198513 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:34.198691 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:34.198851 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:34.199515 (+ 664us) server_negotiation.cc:300] Negotiation successful
0131 03:03:34.199693 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260131 03:03:34.200595 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.187294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:40445 (local address 127.6.22.66:50099)
0131 03:03:34.187756 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:34.187800 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:34.188608 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:34.189358 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:34.189366 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:34.189786 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:34.190866 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.190882 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.191787 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.191790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:34.192201 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.192209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.192509 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.193245 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:34.193283 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:34.194919 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:34.197140 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:34.197172 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:34.197196 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:34.197782 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:34.198181 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:34.198188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:34.198193 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:34.198390 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:34.198892 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:34.198902 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:34.199225 (+ 323us) client_negotiation.cc:769] Sending connection context
0131 03:03:34.199480 (+ 255us) client_negotiation.cc:240] Negotiation successful
0131 03:03:34.199808 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":136,"threads_started":1}
I20260131 03:03:34.202248 17501 heartbeater.cc:344] Connected to a master server at 127.6.22.126:40445
I20260131 03:03:34.202543 17501 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
I20260131 03:03:34.203146 17501 heartbeater.cc:507] Master 127.6.22.126:40445 requested a full tablet report, sending...
I20260131 03:03:34.204433 17170 ts_manager.cc:194] Registered new tserver with Master: 849ed8010cf5476b8823a224f9f54805 (127.6.22.66:41451)
I20260131 03:03:34.205124 17170 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:50099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:03:34.247929 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:40445
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:38189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:03:34.358888 17508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:03:34.359176 17508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:03:34.359246 17508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:03:34.362905 17508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:03:34.362995 17508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:03:34.363126 17508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:03:34.367950 17508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:38189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:03:34.369184 17508 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:03:34.370100 17508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:03:34.377019 17514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:34.377029 17516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:03:34.377024 17513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:03:34.377424 17508 server_base.cc:1060] running on GCE node
I20260131 03:03:34.377861 17508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:03:34.378535 17508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:03:34.379743 17508 hybrid_clock.cc:648] HybridClock initialized: now 1769828614379713 us; error 44 us; skew 500 ppm
Jan 31 03:03:34 dist-test-slave-j368 krb5kdc[17124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828614, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:03:34.382776 17508 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:03:34.384070 17508 webserver.cc:492] Webserver started at http://127.6.22.67:38137/ using document root <none> and password file <none>
I20260131 03:03:34.384766 17508 fs_manager.cc:362] Metadata directory not provided
I20260131 03:03:34.384846 17508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:03:34.385056 17508 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:03:34.386943 17508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "06f780f1ddc446f09d7581e1498061a4"
format_stamp: "Formatted at 2026-01-31 03:03:34 on dist-test-slave-j368"
server_key: "c2a2176a58e06d0e8dfa4da91fa07556"
server_key_iv: "a954e97fb2c2545027d30d9788a29ef4"
server_key_version: "encryptionkey@0"
I20260131 03:03:34.387503 17508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "06f780f1ddc446f09d7581e1498061a4"
format_stamp: "Formatted at 2026-01-31 03:03:34 on dist-test-slave-j368"
server_key: "c2a2176a58e06d0e8dfa4da91fa07556"
server_key_iv: "a954e97fb2c2545027d30d9788a29ef4"
server_key_version: "encryptionkey@0"
I20260131 03:03:34.391376 17508 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260131 03:03:34.394102 17523 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:34.395274 17508 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:03:34.395545 17508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "06f780f1ddc446f09d7581e1498061a4"
format_stamp: "Formatted at 2026-01-31 03:03:34 on dist-test-slave-j368"
server_key: "c2a2176a58e06d0e8dfa4da91fa07556"
server_key_iv: "a954e97fb2c2545027d30d9788a29ef4"
server_key_version: "encryptionkey@0"
I20260131 03:03:34.395676 17508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:03:34.415973 17508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:03:34.419257 17508 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:03:34.419492 17508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:03:34.420094 17508 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:03:34.421137 17508 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:03:34.421217 17508 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:34.421286 17508 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:03:34.421330 17508 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:03:34.431679 17508 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:46807
I20260131 03:03:34.431710 17636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:46807 every 8 connection(s)
I20260131 03:03:34.432825 17508 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:03:34.434990 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 17508
I20260131 03:03:34.435141 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:03:34.435428 6233 external_mini_cluster.cc:1468] Setting key e8883d4072ca4724a7d06783358a5f7c
Jan 31 03:03:34 dist-test-slave-j368 krb5kdc[17124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828614, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:03:34.446866 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.434780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:57485 (local address 127.6.22.126:40445)
0131 03:03:34.434982 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:34.434987 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:34.435774 (+ 787us) server_negotiation.cc:408] Connection header received
0131 03:03:34.436835 (+ 1061us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:34.436838 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:34.436899 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:34.436996 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:34.438964 (+ 1968us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.439487 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.440236 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.440448 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.443031 (+ 2583us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:34.443064 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:34.443066 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:34.443104 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:34.444611 (+ 1507us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:34.445319 (+ 708us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:34.445323 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:34.445324 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:34.445374 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:34.445705 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:34.445708 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:34.445709 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:34.445918 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:34.446008 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:34.446574 (+ 566us) server_negotiation.cc:300] Negotiation successful
0131 03:03:34.446700 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260131 03:03:34.447584 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.435096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:40445 (local address 127.6.22.67:57485)
0131 03:03:34.435615 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:34.435663 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:34.436582 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:34.437205 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:34.437217 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:34.437775 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:03:34.438787 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.438803 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.439600 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.439603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:34.440009 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.440016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.440214 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.440970 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:03:34.441009 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:03:34.442779 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:03:34.444839 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:34.444845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:34.444856 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:34.445177 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:34.445479 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:03:34.445481 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:03:34.445483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:03:34.445591 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:03:34.446030 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:03:34.446035 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:03:34.446318 (+ 283us) client_negotiation.cc:769] Sending connection context
0131 03:03:34.446515 (+ 197us) client_negotiation.cc:240] Negotiation successful
0131 03:03:34.446773 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":128,"threads_started":1}
I20260131 03:03:34.449036 17637 heartbeater.cc:344] Connected to a master server at 127.6.22.126:40445
I20260131 03:03:34.449333 17637 heartbeater.cc:461] Registering TS with master...
I20260131 03:03:34.450016 17637 heartbeater.cc:507] Master 127.6.22.126:40445 requested a full tablet report, sending...
I20260131 03:03:34.451184 17170 ts_manager.cc:194] Registered new tserver with Master: 06f780f1ddc446f09d7581e1498061a4 (127.6.22.67:46807)
I20260131 03:03:34.451768 17170 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:57485
I20260131 03:03:34.462704 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:03:34.472865 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.464902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.6.22.126:40445)
0131 03:03:34.465260 (+ 358us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:34.465264 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:34.465279 (+ 15us) server_negotiation.cc:408] Connection header received
0131 03:03:34.465492 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:34.465497 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:34.465570 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:34.465682 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:03:34.466692 (+ 1010us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.467229 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.467995 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.468198 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.469438 (+ 1240us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:03:34.469462 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:03:34.469465 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:03:34.469500 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:03:34.471071 (+ 1571us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:34.471651 (+ 580us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:34.471654 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:34.471656 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:34.471701 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:03:34.471907 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:03:34.471909 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:03:34.471910 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:03:34.472119 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:03:34.472238 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:34.472467 (+ 229us) server_negotiation.cc:300] Negotiation successful
0131 03:03:34.472581 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260131 03:03:34.475775 17170 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:03:34.478224 17170 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:03:34.493891 17650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.489150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:38501 (local address 127.0.0.1:51758)
0131 03:03:34.490161 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:34.490174 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:34.490264 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:34.490818 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:34.490824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:34.490845 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:34.491174 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.491185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.492386 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.492389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:34.493289 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.493296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.493418 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.493494 (+ 76us) client_negotiation.cc:769] Sending connection context
0131 03:03:34.493589 (+ 95us) client_negotiation.cc:240] Negotiation successful
0131 03:03:34.493698 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":129,"threads_started":1}
I20260131 03:03:34.494616 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.489689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.6.22.65:38501)
0131 03:03:34.490257 (+ 568us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:34.490263 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:34.490289 (+ 26us) server_negotiation.cc:408] Connection header received
0131 03:03:34.490438 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:34.490444 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:34.490657 (+ 213us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:34.490808 (+ 151us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:34.491365 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.492239 (+ 874us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.493408 (+ 1169us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.493952 (+ 544us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.494253 (+ 301us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:34.494327 (+ 74us) server_negotiation.cc:300] Negotiation successful
0131 03:03:34.494456 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":103,"threads_started":1}
I20260131 03:03:34.494603 17648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.489508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:41451 (local address 127.0.0.1:44848)
0131 03:03:34.490492 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:34.490509 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:34.490634 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:34.491107 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:34.491110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:34.491127 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:34.491378 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.491384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.492999 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.493003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:34.494095 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.494106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.494233 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.494261 (+ 28us) client_negotiation.cc:769] Sending connection context
0131 03:03:34.494318 (+ 57us) client_negotiation.cc:240] Negotiation successful
0131 03:03:34.494394 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":106,"threads_started":1}
I20260131 03:03:34.495462 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.489067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.6.22.67:46807)
0131 03:03:34.490059 (+ 992us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:34.490070 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:34.490417 (+ 347us) server_negotiation.cc:408] Connection header received
0131 03:03:34.490509 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:34.490514 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:34.490772 (+ 258us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:34.490934 (+ 162us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:34.491632 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.492849 (+ 1217us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.494511 (+ 1662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.495081 (+ 570us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.495204 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:34.495269 (+ 65us) server_negotiation.cc:300] Negotiation successful
0131 03:03:34.495350 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":175,"threads_started":1}
I20260131 03:03:34.495850 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.489136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44848 (local address 127.6.22.66:41451)
0131 03:03:34.490334 (+ 1198us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:34.490342 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:34.490627 (+ 285us) server_negotiation.cc:408] Connection header received
0131 03:03:34.490781 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:34.490789 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:34.490940 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:34.491091 (+ 151us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:34.491551 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.492843 (+ 1292us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.494655 (+ 1812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.495409 (+ 754us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.495513 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:34.495583 (+ 70us) server_negotiation.cc:300] Negotiation successful
0131 03:03:34.495692 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1072,"thread_start_us":157,"threads_started":1}
I20260131 03:03:34.496085 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.488724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:46807 (local address 127.0.0.1:47306)
0131 03:03:34.490008 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:34.490073 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:34.490265 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:34.491154 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:34.491157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:34.491177 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:34.491487 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.491495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.493002 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.493005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:34.494095 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.494106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.494233 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.494261 (+ 28us) client_negotiation.cc:769] Sending connection context
0131 03:03:34.494318 (+ 57us) client_negotiation.cc:240] Negotiation successful
0131 03:03:34.494393 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"thread_start_us":137,"threads_started":1}
I20260131 03:03:34.498307 17299 tablet_service.cc:1505] Processing CreateTablet for tablet 55042b8f61cc44019365ec3e5475da01 (DEFAULT_TABLE table=test-table [id=c853de17d21d430cb14274ec42a83f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:34.498698 17435 tablet_service.cc:1505] Processing CreateTablet for tablet 55042b8f61cc44019365ec3e5475da01 (DEFAULT_TABLE table=test-table [id=c853de17d21d430cb14274ec42a83f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:34.499473 17299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55042b8f61cc44019365ec3e5475da01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:34.499238 17571 tablet_service.cc:1505] Processing CreateTablet for tablet 55042b8f61cc44019365ec3e5475da01 (DEFAULT_TABLE table=test-table [id=c853de17d21d430cb14274ec42a83f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:03:34.499890 17435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55042b8f61cc44019365ec3e5475da01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:34.500279 17571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55042b8f61cc44019365ec3e5475da01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:03:34.506940 17654 tablet_bootstrap.cc:492] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805: Bootstrap starting.
I20260131 03:03:34.508605 17655 tablet_bootstrap.cc:492] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28: Bootstrap starting.
I20260131 03:03:34.508950 17656 tablet_bootstrap.cc:492] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4: Bootstrap starting.
I20260131 03:03:34.509774 17654 tablet_bootstrap.cc:654] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:34.510807 17654 log.cc:826] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805: Log is configured to *not* fsync() on all Append() calls
I20260131 03:03:34.511431 17656 tablet_bootstrap.cc:654] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:34.511431 17655 tablet_bootstrap.cc:654] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28: Neither blocks nor log segments found. Creating new log.
I20260131 03:03:34.512444 17656 log.cc:826] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4: Log is configured to *not* fsync() on all Append() calls
I20260131 03:03:34.512442 17655 log.cc:826] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28: Log is configured to *not* fsync() on all Append() calls
I20260131 03:03:34.513100 17654 tablet_bootstrap.cc:492] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805: No bootstrap required, opened a new log
I20260131 03:03:34.513336 17654 ts_tablet_manager.cc:1403] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260131 03:03:34.514712 17656 tablet_bootstrap.cc:492] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4: No bootstrap required, opened a new log
I20260131 03:03:34.514823 17655 tablet_bootstrap.cc:492] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28: No bootstrap required, opened a new log
I20260131 03:03:34.514945 17656 ts_tablet_manager.cc:1403] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260131 03:03:34.515095 17655 ts_tablet_manager.cc:1403] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260131 03:03:34.517807 17654 raft_consensus.cc:359] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f780f1ddc446f09d7581e1498061a4" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 46807 } } peers { permanent_uuid: "849ed8010cf5476b8823a224f9f54805" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41451 } } peers { permanent_uuid: "8478165796104a27a780d4d1eeef7c28" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 38501 } }
I20260131 03:03:34.518118 17654 raft_consensus.cc:385] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:34.518173 17654 raft_consensus.cc:740] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849ed8010cf5476b8823a224f9f54805, State: Initialized, Role: FOLLOWER
I20260131 03:03:34.518133 17655 raft_consensus.cc:359] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f780f1ddc446f09d7581e1498061a4" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 46807 } } peers { permanent_uuid: "849ed8010cf5476b8823a224f9f54805" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41451 } } peers { permanent_uuid: "8478165796104a27a780d4d1eeef7c28" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 38501 } }
I20260131 03:03:34.518414 17655 raft_consensus.cc:385] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:34.518488 17655 raft_consensus.cc:740] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8478165796104a27a780d4d1eeef7c28, State: Initialized, Role: FOLLOWER
I20260131 03:03:34.518814 17654 consensus_queue.cc:260] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f780f1ddc446f09d7581e1498061a4" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 46807 } } peers { permanent_uuid: "849ed8010cf5476b8823a224f9f54805" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41451 } } peers { permanent_uuid: "8478165796104a27a780d4d1eeef7c28" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 38501 } }
I20260131 03:03:34.519074 17655 consensus_queue.cc:260] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f780f1ddc446f09d7581e1498061a4" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 46807 } } peers { permanent_uuid: "849ed8010cf5476b8823a224f9f54805" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41451 } } peers { permanent_uuid: "8478165796104a27a780d4d1eeef7c28" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 38501 } }
I20260131 03:03:34.519079 17656 raft_consensus.cc:359] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f780f1ddc446f09d7581e1498061a4" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 46807 } } peers { permanent_uuid: "849ed8010cf5476b8823a224f9f54805" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41451 } } peers { permanent_uuid: "8478165796104a27a780d4d1eeef7c28" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 38501 } }
I20260131 03:03:34.519311 17656 raft_consensus.cc:385] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:03:34.519373 17656 raft_consensus.cc:740] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f780f1ddc446f09d7581e1498061a4, State: Initialized, Role: FOLLOWER
I20260131 03:03:34.519944 17501 heartbeater.cc:499] Master 127.6.22.126:40445 was elected leader, sending a full tablet report...
I20260131 03:03:34.520177 17365 heartbeater.cc:499] Master 127.6.22.126:40445 was elected leader, sending a full tablet report...
I20260131 03:03:34.520188 17655 ts_tablet_manager.cc:1434] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260131 03:03:34.520884 17656 consensus_queue.cc:260] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f780f1ddc446f09d7581e1498061a4" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 46807 } } peers { permanent_uuid: "849ed8010cf5476b8823a224f9f54805" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41451 } } peers { permanent_uuid: "8478165796104a27a780d4d1eeef7c28" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 38501 } }
I20260131 03:03:34.519951 17654 ts_tablet_manager.cc:1434] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260131 03:03:34.521911 17637 heartbeater.cc:499] Master 127.6.22.126:40445 was elected leader, sending a full tablet report...
I20260131 03:03:34.522007 17656 ts_tablet_manager.cc:1434] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260131 03:03:34.646436 17662 raft_consensus.cc:493] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:03:34.646735 17662 raft_consensus.cc:515] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f780f1ddc446f09d7581e1498061a4" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 46807 } } peers { permanent_uuid: "849ed8010cf5476b8823a224f9f54805" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41451 } } peers { permanent_uuid: "8478165796104a27a780d4d1eeef7c28" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 38501 } }
I20260131 03:03:34.648272 17662 leader_election.cc:290] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 849ed8010cf5476b8823a224f9f54805 (127.6.22.66:41451), 8478165796104a27a780d4d1eeef7c28 (127.6.22.65:38501)
I20260131 03:03:34.652459 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.648629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:38501 (local address 127.6.22.67:41111)
0131 03:03:34.648834 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:34.648856 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:34.649031 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:34.649407 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:34.649411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:34.649434 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:34.649785 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.649794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.650889 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.650894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:34.651879 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.651893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.652090 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.652117 (+ 27us) client_negotiation.cc:769] Sending connection context
0131 03:03:34.652190 (+ 73us) client_negotiation.cc:240] Negotiation successful
0131 03:03:34.652259 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260131 03:03:34.652474 17663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.648629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:41451 (local address 127.6.22.67:58829)
0131 03:03:34.649422 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:34.649436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:03:34.649545 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:03:34.649812 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:03:34.649816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:03:34.649840 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:03:34.650126 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.650131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.651126 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.651130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:03:34.651981 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:03:34.651990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.652090 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.652117 (+ 27us) client_negotiation.cc:769] Sending connection context
0131 03:03:34.652190 (+ 73us) client_negotiation.cc:240] Negotiation successful
0131 03:03:34.652262 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":362,"thread_start_us":164,"threads_started":1}
I20260131 03:03:34.652954 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.648812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:41111 (local address 127.6.22.65:38501)
0131 03:03:34.649019 (+ 207us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:34.649025 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:34.649052 (+ 27us) server_negotiation.cc:408] Connection header received
0131 03:03:34.649167 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:34.649184 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:34.649259 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:34.649374 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:34.649913 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.650639 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.652096 (+ 1457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.652637 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.652704 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:34.652757 (+ 53us) server_negotiation.cc:300] Negotiation successful
0131 03:03:34.652823 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260131 03:03:34.652967 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.648911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:58829 (local address 127.6.22.66:41451)
0131 03:03:34.649132 (+ 221us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:34.649140 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:34.649557 (+ 417us) server_negotiation.cc:408] Connection header received
0131 03:03:34.649639 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:34.649641 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:34.649687 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:34.649751 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:03:34.650262 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.650947 (+ 685us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.652104 (+ 1157us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.652631 (+ 527us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.652698 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:34.652757 (+ 59us) server_negotiation.cc:300] Negotiation successful
0131 03:03:34.652823 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260131 03:03:34.653733 17319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55042b8f61cc44019365ec3e5475da01" candidate_uuid: "06f780f1ddc446f09d7581e1498061a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8478165796104a27a780d4d1eeef7c28" is_pre_election: true
I20260131 03:03:34.653733 17455 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55042b8f61cc44019365ec3e5475da01" candidate_uuid: "06f780f1ddc446f09d7581e1498061a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849ed8010cf5476b8823a224f9f54805" is_pre_election: true
I20260131 03:03:34.654053 17319 raft_consensus.cc:2468] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f780f1ddc446f09d7581e1498061a4 in term 0.
I20260131 03:03:34.654053 17455 raft_consensus.cc:2468] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f780f1ddc446f09d7581e1498061a4 in term 0.
I20260131 03:03:34.654613 17527 leader_election.cc:304] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06f780f1ddc446f09d7581e1498061a4, 8478165796104a27a780d4d1eeef7c28, 849ed8010cf5476b8823a224f9f54805; no voters:
I20260131 03:03:34.654884 17662 raft_consensus.cc:2804] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:03:34.654961 17662 raft_consensus.cc:493] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:03:34.655004 17662 raft_consensus.cc:3060] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:34.656031 17662 raft_consensus.cc:515] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f780f1ddc446f09d7581e1498061a4" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 46807 } } peers { permanent_uuid: "849ed8010cf5476b8823a224f9f54805" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41451 } } peers { permanent_uuid: "8478165796104a27a780d4d1eeef7c28" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 38501 } }
I20260131 03:03:34.656471 17662 leader_election.cc:290] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [CANDIDATE]: Term 1 election: Requested vote from peers 849ed8010cf5476b8823a224f9f54805 (127.6.22.66:41451), 8478165796104a27a780d4d1eeef7c28 (127.6.22.65:38501)
I20260131 03:03:34.656910 17455 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55042b8f61cc44019365ec3e5475da01" candidate_uuid: "06f780f1ddc446f09d7581e1498061a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849ed8010cf5476b8823a224f9f54805"
I20260131 03:03:34.657079 17455 raft_consensus.cc:3060] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:34.657107 17319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55042b8f61cc44019365ec3e5475da01" candidate_uuid: "06f780f1ddc446f09d7581e1498061a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8478165796104a27a780d4d1eeef7c28"
I20260131 03:03:34.657281 17319 raft_consensus.cc:3060] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:03:34.658277 17455 raft_consensus.cc:2468] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f780f1ddc446f09d7581e1498061a4 in term 1.
I20260131 03:03:34.658634 17527 leader_election.cc:304] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f780f1ddc446f09d7581e1498061a4, 849ed8010cf5476b8823a224f9f54805; no voters:
I20260131 03:03:34.658869 17662 raft_consensus.cc:2804] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:03:34.658946 17319 raft_consensus.cc:2468] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f780f1ddc446f09d7581e1498061a4 in term 1.
I20260131 03:03:34.659142 17662 raft_consensus.cc:697] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 1 LEADER]: Becoming Leader. State: Replica: 06f780f1ddc446f09d7581e1498061a4, State: Running, Role: LEADER
I20260131 03:03:34.659526 17662 consensus_queue.cc:237] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f780f1ddc446f09d7581e1498061a4" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 46807 } } peers { permanent_uuid: "849ed8010cf5476b8823a224f9f54805" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41451 } } peers { permanent_uuid: "8478165796104a27a780d4d1eeef7c28" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 38501 } }
I20260131 03:03:34.663079 17170 catalog_manager.cc:5654] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06f780f1ddc446f09d7581e1498061a4 (127.6.22.67). New cstate: current_term: 1 leader_uuid: "06f780f1ddc446f09d7581e1498061a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f780f1ddc446f09d7581e1498061a4" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 46807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849ed8010cf5476b8823a224f9f54805" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8478165796104a27a780d4d1eeef7c28" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 38501 } health_report { overall_health: UNKNOWN } } }
W20260131 03:03:34.683912 17638 tablet.cc:2378] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:03:34.686436 17502 tablet.cc:2378] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:03:34.688076 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.684377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.6.22.67:46807)
0131 03:03:34.684529 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0131 03:03:34.684533 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:03:34.684628 (+ 95us) server_negotiation.cc:408] Connection header received
0131 03:03:34.684907 (+ 279us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:03:34.684909 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:03:34.684975 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:03:34.685074 (+ 99us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:03:34.685464 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.685969 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:03:34.686883 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:03:34.687040 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:03:34.687149 (+ 109us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:03:34.687637 (+ 488us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:03:34.687743 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0131 03:03:34.687902 (+ 159us) server_negotiation.cc:300] Negotiation successful
0131 03:03:34.687958 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260131 03:03:34.689543 17366 tablet.cc:2378] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:03:34.695469 17319 raft_consensus.cc:1275] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28 [term 1 FOLLOWER]: Refusing update from remote peer 06f780f1ddc446f09d7581e1498061a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:34.695470 17455 raft_consensus.cc:1275] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805 [term 1 FOLLOWER]: Refusing update from remote peer 06f780f1ddc446f09d7581e1498061a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:03:34.696425 17664 consensus_queue.cc:1048] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849ed8010cf5476b8823a224f9f54805" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:34.696638 17662 consensus_queue.cc:1048] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8478165796104a27a780d4d1eeef7c28" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 38501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:03:34.704634 17668 mvcc.cc:204] Tried to move back new op lower bound from 7249218005783490560 to 7249218005648502784. Current Snapshot: MvccSnapshot[applied={T|T < 7249218005783490560}]
I20260131 03:03:34.714843 17170 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:03:34.715082 17170 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:03:34.718127 17170 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e118088fc49e4ccd9c4837034aeaf217: Sending DeleteTablet for 3 replicas of tablet 55042b8f61cc44019365ec3e5475da01
I20260131 03:03:34.719062 17299 tablet_service.cc:1552] Processing DeleteTablet for tablet 55042b8f61cc44019365ec3e5475da01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:34 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:51758
I20260131 03:03:34.719062 17571 tablet_service.cc:1552] Processing DeleteTablet for tablet 55042b8f61cc44019365ec3e5475da01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:34 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:47306
I20260131 03:03:34.719125 17435 tablet_service.cc:1552] Processing DeleteTablet for tablet 55042b8f61cc44019365ec3e5475da01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:03:34 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:44848
I20260131 03:03:34.719660 17679 tablet_replica.cc:333] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805: stopping tablet replica
I20260131 03:03:34.719918 17678 tablet_replica.cc:333] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4: stopping tablet replica
I20260131 03:03:34.719982 17679 raft_consensus.cc:2243] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:03:34.720176 17677 tablet_replica.cc:333] T 55042b8f61cc44019365ec3e5475da01 P 8478165796104a27a780d4d1eeef7c28: stopping tablet replica
I20260131 03:03:34.720477 17678 raft_consensus.cc:2243] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:03:34.720589 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 17236
I20260131 03:03:34.721613 17678 raft_consensus.cc:2272] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:34.720391 17679 raft_consensus.cc:2272] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:03:34.723358 17678 ts_tablet_manager.cc:1916] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:34.723716 17679 ts_tablet_manager.cc:1916] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:03:34.726516 17679 ts_tablet_manager.cc:1929] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:03:34.726643 17679 log.cc:1199] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/55042b8f61cc44019365ec3e5475da01
I20260131 03:03:34.726930 17678 ts_tablet_manager.cc:1929] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:03:34.727010 17679 ts_tablet_manager.cc:1950] T 55042b8f61cc44019365ec3e5475da01 P 849ed8010cf5476b8823a224f9f54805: Deleting consensus metadata
I20260131 03:03:34.727164 17678 log.cc:1199] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/55042b8f61cc44019365ec3e5475da01
I20260131 03:03:34.727571 17678 ts_tablet_manager.cc:1950] T 55042b8f61cc44019365ec3e5475da01 P 06f780f1ddc446f09d7581e1498061a4: Deleting consensus metadata
I20260131 03:03:34.728468 17159 catalog_manager.cc:4985] TS 849ed8010cf5476b8823a224f9f54805 (127.6.22.66:41451): tablet 55042b8f61cc44019365ec3e5475da01 (table test-table [id=c853de17d21d430cb14274ec42a83f3a]) successfully deleted
I20260131 03:03:34.728853 17157 catalog_manager.cc:4985] TS 06f780f1ddc446f09d7581e1498061a4 (127.6.22.67:46807): tablet 55042b8f61cc44019365ec3e5475da01 (table test-table [id=c853de17d21d430cb14274ec42a83f3a]) successfully deleted
W20260131 03:03:34.730199 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.22.65:38501 (error 108)
I20260131 03:03:34.731011 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 17372
I20260131 03:03:34.731369 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:03:34.731088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:38501 (local address 127.0.0.1:51760)
0131 03:03:34.731218 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0131 03:03:34.731283 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.65:38501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260131 03:03:34.733150 17158 catalog_manager.cc:4712] TS 8478165796104a27a780d4d1eeef7c28 (127.6.22.65:38501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55042b8f61cc44019365ec3e5475da01: Network error: Client connection negotiation failed: client connection to 127.6.22.65:38501: connect: Connection refused (error 111)
I20260131 03:03:34.737802 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 17508
I20260131 03:03:34.743909 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 17140
2026-01-31T03:03:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3457 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:03:34 dist-test-slave-j368 krb5kdc[17683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:03:34 dist-test-slave-j368 krb5kdc[17683](info): set up 2 sockets
Jan 31 03:03:34 dist-test-slave-j368 krb5kdc[17683](info): commencing operation
krb5kdc: starting...
W20260131 03:03:36.763265 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.996s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:03:36 dist-test-slave-j368 krb5kdc[17683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:03:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:03:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.22.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.22.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/rangeradmin_127.6.22.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/rangeradmin_127.6.22.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.22.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.22.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/rangerlookup_127.6.22.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/rangerlookup_127.6.22.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.22.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/HTTP_127.6.22.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/HTTP_127.6.22.84@KRBTEST.COM.keytab.
I20260131 03:03:36.942018 6233 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMlJawA/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/postgres -l logfile start
2026-01-31 03:03:39.490 UTC [17714] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-01-31 03:03:39.490 UTC [17714] LOG: listening on IPv4 address "127.6.22.84", port 51525
2026-01-31 03:03:39.493 UTC [17714] LOG: listening on Unix socket "/tmp/.s.PGSQL.51525"
2026-01-31 03:03:39.498 UTC [17719] LOG: database system was shut down at 2026-01-31 03:03:38 UTC
2026-01-31 03:03:39.503 UTC [17714] LOG: database system is ready to accept connections
I20260131 03:03:41.422971 6233 mini_postgres.cc:96] Postgres bound to 51525
2026-01-31 03:03:41.429 UTC [17725] FATAL: database "slave" does not exist
127.6.22.84:51525 - accepting connections
I20260131 03:03:41.430523 6233 mini_ranger.cc:161] Starting Ranger...
I20260131 03:03:41.450495 6233 mini_ranger.cc:84] Created miniranger Postgres user
I20260131 03:03:41.520907 6233 mini_ranger.cc:87] Created ranger Postgres database
I20260131 03:03:41.521041 6233 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-admin
2026-01-31 03:03:41,817 [I] DB FLAVOR :POSTGRES
2026-01-31 03:03:41,818 [I] --------- Verifying Ranger DB connection ---------
2026-01-31 03:03:41,818 [I] Checking connection..
2026-01-31 03:03:41,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-01-31 03:03:42,164 [I] Checking connection passed.
2026-01-31 03:03:42,165 [I] --------- Verifying version history table ---------
2026-01-31 03:03:42,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-31 03:03:42,490 [I] Table x_db_version_h does not exist in database ranger
2026-01-31 03:03:42,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-31 03:03:42,816 [I] Table x_db_version_h does not exist in database ranger
2026-01-31 03:03:42,816 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-01-31 03:03:42,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-01-31 03:03:43.132 UTC [17792] WARNING: there is no transaction in progress
2026-01-31 03:03:43,140 [I] create_dbversion_catalog.sql file imported successfully
2026-01-31 03:03:43,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-31 03:03:43,462 [I] Table x_db_version_h already exists in database 'ranger'
2026-01-31 03:03:43,462 [I] --------- Importing Ranger Core DB Schema ---------
2026-01-31 03:03:43,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-01-31 03:03:43,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-01-31 03:03:44,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j368.c.gcp-upstream.internal','N') ;"
2026-01-31 03:03:44,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-01-31 03:03:44,781 [I] Table x_portal_user does not exist in database ranger
2026-01-31 03:03:44,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-01-31 03:03:45,111 [I] Table x_policy_ref_group does not exist in database ranger
2026-01-31 03:03:45,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-01-31 03:03:45,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-01-31 03:03:45,769 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-01-31 03:03:45,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-01-31 03:03:46.717 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.732 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.749 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.764 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.780 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.795 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.861 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.867 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.878 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.889 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.900 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.911 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.920 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.925 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.929 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:46.934 UTC [17909] WARNING: there is no transaction in progress
2026-01-31 03:03:47,441 [I] ranger_core_db_postgres.sql file imported successfully
2026-01-31 03:03:47,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-01-31 03:03:47,768 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-01-31 03:03:47,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-01-31 03:03:48,102 [I] Table x_portal_user already exists in database 'ranger'
2026-01-31 03:03:48,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-01-31 03:03:48,463 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-01-31 03:03:48,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-01-31 03:03:48,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-01-31 03:03:49,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j368.c.gcp-upstream.internal';"
2026-01-31 03:03:49,410 [I] CORE_DB_SCHEMA import status has been updated
2026-01-31 03:03:49,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-01-31 03:03:49,739 [I] DB_PATCHES have already been applied
I20260131 03:03:49.745654 6233 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMlJawA/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260131 03:03:49.745738 6233 mini_ranger.cc:193] Using host: 127.6.22.84
I20260131 03:03:49.748952 6233 mini_ranger.cc:227] Ranger admin URL: http://127.6.22.84:38209
Jan 31, 2026 3:03:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 31, 2026 3:03:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 31, 2026 3:03:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-admin/ews/webapp
Jan 31, 2026 3:03:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jan 31, 2026 3:03:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-admin/ews/webapp] .....
Jan 31, 2026 3:03:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-admin/ews/webapp].
Jan 31, 2026 3:03:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260131 03:03:52.055251 6233 mini_ranger.cc:160] Time spent starting Ranger: real 10.625s user 0.007s sys 0.004s
Jan 31, 2026 3:04:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jan 31, 2026 3:04:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
Jan 31, 2026 3:04:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jan 31, 2026 3:04:14 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jan 31, 2026 3:04:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260131 03:04:15.584474 6233 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.22.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.22.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/rangerkms_127.6.22.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/rangerkms_127.6.22.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.22.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.22.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/HTTP_127.6.22.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.22.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/HTTP_127.6.22.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jan 31 03:04:15 dist-test-slave-j368 krb5kdc[17683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828655, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260131 03:04:15.669978 6233 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260131 03:04:15.680908 6233 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260131 03:04:15.794713 6233 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260131 03:04:15.794865 6233 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-kms
I20260131 03:04:15.794921 6233 mini_ranger_kms.cc:227] Using postgres at 127.6.22.84:51525
2026-01-31 03:04:15,994 [I] DB FLAVOR :POSTGRES
2026-01-31 03:04:15,994 [I] --------- Verifying Ranger DB connection ---------
2026-01-31 03:04:15,994 [I] Checking connection
2026-01-31 03:04:15,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-01-31 03:04:16,327 [I] connection success
2026-01-31 03:04:16,327 [I] --------- Verifying Ranger DB tables ---------
2026-01-31 03:04:16,327 [I] Verifying table ranger_masterkey in database rangerkms
2026-01-31 03:04:16,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-01-31 03:04:16,686 [I] Table ranger_masterkey does not exist in database rangerkms
2026-01-31 03:04:16,686 [I] --------- Importing Ranger Core DB Schema ---------
2026-01-31 03:04:16,686 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-01-31 03:04:16,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:51525/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-01-31 03:04:17,015 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260131 03:04:17.572568 6233 mini_ranger_kms.cc:310] Created kms service
I20260131 03:04:17.823634 6233 mini_ranger_kms.cc:326] Created kudu user
I20260131 03:04:17.869891 6233 mini_ranger_kms.cc:343] Created rangerkms user
Jan 31 03:04:17 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828655, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
I20260131 03:04:18.323009 6233 mini_ranger_kms.cc:384] Added ranger policy
I20260131 03:04:18.323204 6233 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260131 03:04:18.323228 6233 mini_ranger_kms.cc:240] Using host: 127.6.22.84
I20260131 03:04:18.327039 6233 mini_ranger_kms.cc:276] Ranger KMS PID: 18101
I20260131 03:04:18.327183 6233 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.22.84:48827
Jan 31, 2026 3:04:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 31, 2026 3:04:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 31, 2026 3:04:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jan 31, 2026 3:04:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-kms/ews/webapp] .....
Jan 31, 2026 3:04:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-kms/ews/webapp].
Jan 31, 2026 3:04:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260131 03:04:20.583824 6233 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.914s user 0.008s sys 0.172s
I20260131 03:04:20.584023 6233 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260131 03:04:20.584149 6233 mini_ranger_kms.cc:401] 127.6.22.84:48827/kms/v1/keys
Jan 31 03:04:20 dist-test-slave-j368 krb5kdc[17683](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1769828660, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.22.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jan 31 03:04:21 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828660, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.22.84@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
Jan 31 03:04:21 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828660, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.22.84@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
Jan 31 03:04:24 dist-test-slave-j368 krb5kdc[17683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:04:24.283726 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:34645
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:34455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:34645
--ranger_config_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.22.84:48827/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:04:24.400923 18154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:04:24.401262 18154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:04:24.401376 18154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:04:24.404995 18154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:04:24.405102 18154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:04:24.405146 18154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:04:24.405203 18154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:04:24.405264 18154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:04:24.410161 18154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:34455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.22.84:48827/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:34645
--ranger_config_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:34645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:04:24.411439 18154 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:04:24.412364 18154 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:04:24.418803 18159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:04:24.418813 18160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:04:24.418918 18154 server_base.cc:1060] running on GCE node
W20260131 03:04:24.418803 18162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:04:24.419687 18154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:04:24.420815 18154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:04:24.421998 18154 hybrid_clock.cc:648] HybridClock initialized: now 1769828664421988 us; error 39 us; skew 500 ppm
Jan 31 03:04:24 dist-test-slave-j368 krb5kdc[17683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828664, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:04:24.424824 18154 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:04:24.426090 18154 webserver.cc:492] Webserver started at http://127.6.22.126:41065/ using document root <none> and password file <none>
I20260131 03:04:24.426697 18154 fs_manager.cc:362] Metadata directory not provided
I20260131 03:04:24.426749 18154 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:04:24.426970 18154 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 31 03:04:24 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828664, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
I20260131 03:04:24.491331 18154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "689d75d8148642dfb2f24e2955b4285d"
format_stamp: "Formatted at 2026-01-31 03:04:24 on dist-test-slave-j368"
server_key: "62aefaf612a64b15f15beb9f75b42672"
server_key_iv: "73c31a97687701bbad72b44d6c086393"
server_key_version: "kuduclusterkey@0"
I20260131 03:04:24.491896 18154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "689d75d8148642dfb2f24e2955b4285d"
format_stamp: "Formatted at 2026-01-31 03:04:24 on dist-test-slave-j368"
server_key: "62aefaf612a64b15f15beb9f75b42672"
server_key_iv: "73c31a97687701bbad72b44d6c086393"
server_key_version: "kuduclusterkey@0"
I20260131 03:04:24.496011 18154 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260131 03:04:24.517014 18169 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:04:24.518277 18154 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260131 03:04:24.518452 18154 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "689d75d8148642dfb2f24e2955b4285d"
format_stamp: "Formatted at 2026-01-31 03:04:24 on dist-test-slave-j368"
server_key: "62aefaf612a64b15f15beb9f75b42672"
server_key_iv: "73c31a97687701bbad72b44d6c086393"
server_key_version: "kuduclusterkey@0"
I20260131 03:04:24.518579 18154 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:04:24.548045 18154 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:04:24.555074 18154 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:04:24.555328 18154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:04:24.563918 18154 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:34645
I20260131 03:04:24.563917 18221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:34645 every 8 connection(s)
I20260131 03:04:24.565099 18154 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:04:24.568578 18222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:04:24.574268 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 18154
I20260131 03:04:24.574492 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:04:24.574817 6233 external_mini_cluster.cc:1468] Setting key 4884d0dc388c613fdb71c1b55f9e0c58
I20260131 03:04:24.576890 18222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d: Bootstrap starting.
I20260131 03:04:24.579705 18222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d: Neither blocks nor log segments found. Creating new log.
I20260131 03:04:24.580551 18222 log.cc:826] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d: Log is configured to *not* fsync() on all Append() calls
I20260131 03:04:24.582747 18222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d: No bootstrap required, opened a new log
Jan 31 03:04:24 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:04:24.586666 18222 raft_consensus.cc:359] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689d75d8148642dfb2f24e2955b4285d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 34645 } }
I20260131 03:04:24.586941 18222 raft_consensus.cc:385] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:04:24.586992 18222 raft_consensus.cc:740] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 689d75d8148642dfb2f24e2955b4285d, State: Initialized, Role: FOLLOWER
I20260131 03:04:24.587527 18222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689d75d8148642dfb2f24e2955b4285d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 34645 } }
I20260131 03:04:24.587698 18222 raft_consensus.cc:399] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:04:24.587821 18222 raft_consensus.cc:493] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:04:24.587944 18222 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:04:24.589249 18222 raft_consensus.cc:515] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689d75d8148642dfb2f24e2955b4285d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 34645 } }
I20260131 03:04:24.589700 18222 leader_election.cc:304] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 689d75d8148642dfb2f24e2955b4285d; no voters:
I20260131 03:04:24.590073 18222 leader_election.cc:290] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:04:24.590346 18227 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:04:24.590613 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:24.576742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.6.22.126:34645)
0131 03:04:24.577244 (+ 502us) server_negotiation.cc:207] Beginning negotiation
0131 03:04:24.577255 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:04:24.577295 (+ 40us) server_negotiation.cc:408] Connection header received
0131 03:04:24.577941 (+ 646us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:04:24.577958 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:04:24.578279 (+ 321us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:04:24.578632 (+ 353us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:04:24.579654 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:24.580475 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:04:24.581358 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:24.581664 (+ 306us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:24.584079 (+ 2415us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:04:24.584103 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:04:24.584117 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:04:24.584171 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:04:24.586963 (+ 2792us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:04:24.587776 (+ 813us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:04:24.587783 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:04:24.587790 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:04:24.587898 (+ 108us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:04:24.588312 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:04:24.588319 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:04:24.588324 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:04:24.588841 (+ 517us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:04:24.589100 (+ 259us) server_negotiation.cc:1036] Waiting for connection context
0131 03:04:24.589495 (+ 395us) server_negotiation.cc:300] Negotiation successful
0131 03:04:24.589767 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":152,"threads_started":1}
I20260131 03:04:24.590714 18227 raft_consensus.cc:697] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [term 1 LEADER]: Becoming Leader. State: Replica: 689d75d8148642dfb2f24e2955b4285d, State: Running, Role: LEADER
I20260131 03:04:24.591326 18227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689d75d8148642dfb2f24e2955b4285d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 34645 } }
I20260131 03:04:24.591377 18222 sys_catalog.cc:565] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:04:24.593508 18228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "689d75d8148642dfb2f24e2955b4285d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689d75d8148642dfb2f24e2955b4285d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 34645 } } }
I20260131 03:04:24.593708 18228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [sys.catalog]: This master's current role is: LEADER
I20260131 03:04:24.594421 18237 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:04:24.595006 18222 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260131 03:04:24.595069 18229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 689d75d8148642dfb2f24e2955b4285d. Latest consensus state: current_term: 1 leader_uuid: "689d75d8148642dfb2f24e2955b4285d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689d75d8148642dfb2f24e2955b4285d" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 34645 } } }
I20260131 03:04:24.595181 18229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d [sys.catalog]: This master's current role is: LEADER
I20260131 03:04:24.601329 18237 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:04:24.611008 18237 catalog_manager.cc:1357] Generated new cluster ID: 8787e94650a94210b971b189b520a8b8
I20260131 03:04:24.611124 18237 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:04:24.625252 18237 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:04:24.627136 18237 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:04:24.638731 18237 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d: Generated new TSK 0
I20260131 03:04:24.639794 18237 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:04:25.441993 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.22.84:48827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:34645
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:34455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:04:25.589654 18268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:04:25.590044 18268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:04:25.590178 18268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:04:25.594105 18268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:04:25.594208 18268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:04:25.594323 18268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:04:25.599120 18268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:34455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.22.84:48827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:34645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:04:25.600400 18268 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:04:25.601346 18268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:04:25.608569 18276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:04:25.608569 18278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:04:25.609591 18275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:04:25.609891 18268 server_base.cc:1060] running on GCE node
I20260131 03:04:25.610307 18268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:04:25.610985 18268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:04:25.612154 18268 hybrid_clock.cc:648] HybridClock initialized: now 1769828665612129 us; error 51 us; skew 500 ppm
Jan 31 03:04:25 dist-test-slave-j368 krb5kdc[17683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828665, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:04:25.614967 18268 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:04:25.616257 18268 webserver.cc:492] Webserver started at http://127.6.22.65:39627/ using document root <none> and password file <none>
I20260131 03:04:25.616911 18268 fs_manager.cc:362] Metadata directory not provided
I20260131 03:04:25.616986 18268 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:04:25.617228 18268 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 31 03:04:25 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828665, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
I20260131 03:04:25.631364 18268 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "c9716dc4bd7f4010a17153de002ba057"
format_stamp: "Formatted at 2026-01-31 03:04:25 on dist-test-slave-j368"
server_key: "385cc25c16445da387f7bedf83fd3482"
server_key_iv: "06a563d487a0a1fb6c862fa1423136c0"
server_key_version: "kuduclusterkey@0"
I20260131 03:04:25.632524 18268 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "c9716dc4bd7f4010a17153de002ba057"
format_stamp: "Formatted at 2026-01-31 03:04:25 on dist-test-slave-j368"
server_key: "385cc25c16445da387f7bedf83fd3482"
server_key_iv: "06a563d487a0a1fb6c862fa1423136c0"
server_key_version: "kuduclusterkey@0"
I20260131 03:04:25.650310 18268 fs_manager.cc:696] Time spent creating directory manager: real 0.017s user 0.003s sys 0.000s
I20260131 03:04:25.683063 18287 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:04:25.684569 18268 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260131 03:04:25.684749 18268 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "c9716dc4bd7f4010a17153de002ba057"
format_stamp: "Formatted at 2026-01-31 03:04:25 on dist-test-slave-j368"
server_key: "385cc25c16445da387f7bedf83fd3482"
server_key_iv: "06a563d487a0a1fb6c862fa1423136c0"
server_key_version: "kuduclusterkey@0"
I20260131 03:04:25.684898 18268 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:04:25.703758 18268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:04:25.706910 18268 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:04:25.707139 18268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:04:25.707824 18268 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:04:25.708854 18268 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:04:25.708926 18268 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:04:25.709043 18268 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:04:25.709098 18268 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:04:25.721421 18268 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:34899
I20260131 03:04:25.722648 18268 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:04:25.723868 18400 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:34899 every 8 connection(s)
I20260131 03:04:25.728147 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 18268
I20260131 03:04:25.728307 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:04:25.728614 6233 external_mini_cluster.cc:1468] Setting key 1276e8763c6e7789addd94f5a9d71ea8
Jan 31 03:04:25 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828665, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
I20260131 03:04:25.740171 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:25.724846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:38023 (local address 127.6.22.126:34645)
0131 03:04:25.725112 (+ 266us) server_negotiation.cc:207] Beginning negotiation
0131 03:04:25.725115 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:04:25.725911 (+ 796us) server_negotiation.cc:408] Connection header received
0131 03:04:25.726817 (+ 906us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:04:25.726821 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:04:25.726883 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:04:25.726999 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:04:25.728367 (+ 1368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:25.729171 (+ 804us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:04:25.730025 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:25.731862 (+ 1837us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:25.735201 (+ 3339us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:04:25.735232 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:04:25.735239 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:04:25.735280 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:04:25.737417 (+ 2137us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:04:25.738210 (+ 793us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:04:25.738215 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:04:25.738220 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:04:25.738284 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:04:25.738728 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:04:25.738731 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:04:25.738733 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:04:25.738963 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:04:25.739048 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0131 03:04:25.739807 (+ 759us) server_negotiation.cc:300] Negotiation successful
0131 03:04:25.739951 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260131 03:04:25.741015 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:25.725264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:34645 (local address 127.6.22.65:38023)
0131 03:04:25.725764 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0131 03:04:25.725802 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0131 03:04:25.726607 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:04:25.727172 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:04:25.727180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:04:25.727472 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:04:25.728156 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:25.728178 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:25.729321 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:04:25.729325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:04:25.729853 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:25.729863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:25.732196 (+ 2333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:25.732961 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:04:25.733030 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:04:25.734925 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:04:25.737641 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:04:25.737648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:04:25.737665 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:04:25.738065 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:04:25.738428 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:04:25.738432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:04:25.738437 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:04:25.738609 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:04:25.739147 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:04:25.739155 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:04:25.739521 (+ 366us) client_negotiation.cc:769] Sending connection context
0131 03:04:25.739774 (+ 253us) client_negotiation.cc:240] Negotiation successful
0131 03:04:25.740110 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":144,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
I20260131 03:04:25.742630 18401 heartbeater.cc:344] Connected to a master server at 127.6.22.126:34645
I20260131 03:04:25.742930 18401 heartbeater.cc:461] Registering TS with master...
I20260131 03:04:25.743666 18401 heartbeater.cc:507] Master 127.6.22.126:34645 requested a full tablet report, sending...
I20260131 03:04:25.745519 18186 ts_manager.cc:194] Registered new tserver with Master: c9716dc4bd7f4010a17153de002ba057 (127.6.22.65:34899)
I20260131 03:04:25.747136 18186 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:38023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Jan 31 03:04:25 dist-test-slave-j368 krb5kdc[17683](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1769828665, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:04:25.785393 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.22.84:48827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:34645
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:34455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:04:25.911617 18409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:04:25.911904 18409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:04:25.911995 18409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:04:25.915874 18409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:04:25.916082 18409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:04:25.916278 18409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:04:25.921228 18409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:34455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.22.84:48827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:34645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:04:25.922541 18409 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:04:25.923796 18409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:04:25.930687 18415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:04:25.930830 18416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:04:25.931491 18418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:04:25.931849 18409 server_base.cc:1060] running on GCE node
I20260131 03:04:25.932260 18409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:04:25.932964 18409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:04:25.940451 18409 hybrid_clock.cc:648] HybridClock initialized: now 1769828665940363 us; error 109 us; skew 500 ppm
Jan 31 03:04:25 dist-test-slave-j368 krb5kdc[17683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828665, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:04:25.944520 18409 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:04:25.946105 18409 webserver.cc:492] Webserver started at http://127.6.22.66:36529/ using document root <none> and password file <none>
I20260131 03:04:25.946892 18409 fs_manager.cc:362] Metadata directory not provided
I20260131 03:04:25.947054 18409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:04:25.947340 18409 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 31 03:04:25 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828665, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
I20260131 03:04:25.967705 18409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "1e8627c4e6554c94b211d22e2ffad9d3"
format_stamp: "Formatted at 2026-01-31 03:04:25 on dist-test-slave-j368"
server_key: "c484ae6488497248c84d76143612ed53"
server_key_iv: "eb0081f162f040f5bf42bf1c86d499c9"
server_key_version: "kuduclusterkey@0"
I20260131 03:04:25.968509 18409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "1e8627c4e6554c94b211d22e2ffad9d3"
format_stamp: "Formatted at 2026-01-31 03:04:25 on dist-test-slave-j368"
server_key: "c484ae6488497248c84d76143612ed53"
server_key_iv: "eb0081f162f040f5bf42bf1c86d499c9"
server_key_version: "kuduclusterkey@0"
I20260131 03:04:25.973518 18409 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260131 03:04:25.993160 18425 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:04:25.994673 18409 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260131 03:04:25.994827 18409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "1e8627c4e6554c94b211d22e2ffad9d3"
format_stamp: "Formatted at 2026-01-31 03:04:25 on dist-test-slave-j368"
server_key: "c484ae6488497248c84d76143612ed53"
server_key_iv: "eb0081f162f040f5bf42bf1c86d499c9"
server_key_version: "kuduclusterkey@0"
I20260131 03:04:25.994959 18409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:04:26.035140 18409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:04:26.041369 18409 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:04:26.041663 18409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:04:26.042400 18409 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:04:26.043589 18409 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:04:26.043696 18409 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:04:26.043790 18409 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:04:26.043849 18409 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:04:26.058802 18409 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:36849
I20260131 03:04:26.059803 18538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:36849 every 8 connection(s)
I20260131 03:04:26.060269 18409 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:04:26.065627 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 18409
I20260131 03:04:26.065775 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:04:26.066049 6233 external_mini_cluster.cc:1468] Setting key eeae844ea2635862e2675c3e1c38c779
WARNING: no policy specified for kudu/127.6.22.67@KRBTEST.COM; defaulting to no policy
Jan 31 03:04:26 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828665, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.67@KRBTEST.COM" created.
I20260131 03:04:26.092130 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.064166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.66:55707 (local address 127.6.22.126:34645)
0131 03:04:26.064306 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0131 03:04:26.064310 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:04:26.067372 (+ 3062us) server_negotiation.cc:408] Connection header received
0131 03:04:26.068760 (+ 1388us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:04:26.068765 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:04:26.068824 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:04:26.068913 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:04:26.071754 (+ 2841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.072725 (+ 971us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.076046 (+ 3321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.076278 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.084142 (+ 7864us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:04:26.084175 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:04:26.084179 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:04:26.084221 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:04:26.086241 (+ 2020us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:04:26.089775 (+ 3534us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:04:26.089781 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:04:26.089783 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:04:26.089879 (+ 96us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:04:26.090390 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:04:26.090394 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:04:26.090396 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:04:26.090650 (+ 254us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:04:26.090749 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0131 03:04:26.091755 (+ 1006us) server_negotiation.cc:300] Negotiation successful
0131 03:04:26.091944 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260131 03:04:26.094686 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.063078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:34645 (local address 127.6.22.66:55707)
0131 03:04:26.067188 (+ 4110us) negotiation.cc:107] Waiting for socket to connect
0131 03:04:26.067252 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0131 03:04:26.068447 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:04:26.069236 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:04:26.069250 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:04:26.069659 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:04:26.070539 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.070560 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.075119 (+ 4559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.075124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:04:26.075855 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.075873 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.076208 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.078314 (+ 2106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:04:26.078378 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:04:26.083848 (+ 5470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:04:26.086662 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:04:26.086670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:04:26.086687 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:04:26.087118 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:04:26.090037 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:04:26.090041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:04:26.090044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:04:26.090262 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:04:26.090934 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:04:26.090943 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:04:26.091347 (+ 404us) client_negotiation.cc:769] Sending connection context
0131 03:04:26.092762 (+ 1415us) client_negotiation.cc:240] Negotiation successful
0131 03:04:26.093156 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3829,"thread_start_us":147,"threads_started":1}
I20260131 03:04:26.096349 18539 heartbeater.cc:344] Connected to a master server at 127.6.22.126:34645
I20260131 03:04:26.096632 18539 heartbeater.cc:461] Registering TS with master...
I20260131 03:04:26.097366 18539 heartbeater.cc:507] Master 127.6.22.126:34645 requested a full tablet report, sending...
I20260131 03:04:26.098692 18186 ts_manager.cc:194] Registered new tserver with Master: 1e8627c4e6554c94b211d22e2ffad9d3 (127.6.22.66:36849)
I20260131 03:04:26.099485 18186 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.66@KRBTEST.COM'} at 127.6.22.66:55707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:04:26.154129 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.22.84:48827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.67:0
--local_ip_for_outbound_sockets=127.6.22.67
--webserver_interface=127.6.22.67
--webserver_port=0
--tserver_master_addrs=127.6.22.126:34645
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:34455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:04:26 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828665, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
W20260131 03:04:26.314550 18546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:04:26.314911 18546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:04:26.314993 18546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:04:26.319948 18546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:04:26.320082 18546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:04:26.320199 18546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.67
I20260131 03:04:26.327068 18546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:34455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.22.84:48827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.22.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:34645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.67
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:04:26.328573 18546 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:04:26.329772 18546 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:04:26.338617 18554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:04:26.340204 18546 server_base.cc:1060] running on GCE node
W20260131 03:04:26.340497 18551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:04:26.343991 18552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:04:26.344609 18546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:04:26.345453 18546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:04:26.352849 18546 hybrid_clock.cc:648] HybridClock initialized: now 1769828666352758 us; error 111 us; skew 500 ppm
Jan 31 03:04:26 dist-test-slave-j368 krb5kdc[17683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828666, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:04:26.357050 18546 init.cc:377] Logged in from keytab as kudu/127.6.22.67@KRBTEST.COM (short username kudu)
I20260131 03:04:26.358781 18546 webserver.cc:492] Webserver started at http://127.6.22.67:45369/ using document root <none> and password file <none>
I20260131 03:04:26.359580 18546 fs_manager.cc:362] Metadata directory not provided
I20260131 03:04:26.359670 18546 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:04:26.359923 18546 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 31 03:04:26 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828666, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
I20260131 03:04:26.377774 18546 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/data/instance:
uuid: "7a133297d4a84b8380e9bfbac28f870b"
format_stamp: "Formatted at 2026-01-31 03:04:26 on dist-test-slave-j368"
server_key: "df299e3cac014e443d51951588c8d245"
server_key_iv: "b5a1a0d9db2bccd1432186e8fe901ebd"
server_key_version: "kuduclusterkey@0"
I20260131 03:04:26.378722 18546 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance:
uuid: "7a133297d4a84b8380e9bfbac28f870b"
format_stamp: "Formatted at 2026-01-31 03:04:26 on dist-test-slave-j368"
server_key: "df299e3cac014e443d51951588c8d245"
server_key_iv: "b5a1a0d9db2bccd1432186e8fe901ebd"
server_key_version: "kuduclusterkey@0"
I20260131 03:04:26.388516 18546 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.000s sys 0.004s
I20260131 03:04:26.408005 18561 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:04:26.409550 18546 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260131 03:04:26.409706 18546 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/wal
uuid: "7a133297d4a84b8380e9bfbac28f870b"
format_stamp: "Formatted at 2026-01-31 03:04:26 on dist-test-slave-j368"
server_key: "df299e3cac014e443d51951588c8d245"
server_key_iv: "b5a1a0d9db2bccd1432186e8fe901ebd"
server_key_version: "kuduclusterkey@0"
I20260131 03:04:26.409852 18546 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:04:26.427724 18546 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:04:26.431068 18546 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:04:26.431317 18546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:04:26.431890 18546 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:04:26.432971 18546 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:04:26.433044 18546 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:04:26.433145 18546 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:04:26.433193 18546 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:04:26.442965 18546 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.67:44525
I20260131 03:04:26.444352 18546 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/data/info.pb
I20260131 03:04:26.444741 18674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.67:44525 every 8 connection(s)
I20260131 03:04:26.449460 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 18546
I20260131 03:04:26.449617 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/wal/instance
I20260131 03:04:26.449904 6233 external_mini_cluster.cc:1468] Setting key f503b416862b646e177bbf3fa2e2f86f
Jan 31 03:04:26 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828666, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.67@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:04:26.465178 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.447762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.67:53649 (local address 127.6.22.126:34645)
0131 03:04:26.447911 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0131 03:04:26.447916 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:04:26.448455 (+ 539us) server_negotiation.cc:408] Connection header received
0131 03:04:26.449772 (+ 1317us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:04:26.449777 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:04:26.449848 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:04:26.449903 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:04:26.453044 (+ 3141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.453741 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.454424 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.454624 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.457049 (+ 2425us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:04:26.457078 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:04:26.457081 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:04:26.457114 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:04:26.458824 (+ 1710us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:04:26.459426 (+ 602us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:04:26.459429 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:04:26.459430 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:04:26.459477 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:04:26.461179 (+ 1702us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:04:26.461183 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:04:26.461185 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:04:26.461437 (+ 252us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:04:26.461541 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0131 03:04:26.464830 (+ 3289us) server_negotiation.cc:300] Negotiation successful
0131 03:04:26.465009 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260131 03:04:26.467130 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.446698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:34645 (local address 127.6.22.67:53649)
0131 03:04:26.448291 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0131 03:04:26.448340 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0131 03:04:26.449479 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:04:26.450725 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:04:26.450838 (+ 113us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:04:26.452174 (+ 1336us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:04:26.452879 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.452896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.453891 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.453894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:04:26.454297 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.454304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.454590 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.455302 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:04:26.455333 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:04:26.456760 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:04:26.458985 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:04:26.458992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:04:26.459007 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:04:26.459299 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:04:26.459580 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:04:26.459584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:04:26.459647 (+ 63us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:04:26.461014 (+ 1367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:04:26.463901 (+ 2887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:04:26.463911 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:04:26.464429 (+ 518us) client_negotiation.cc:769] Sending connection context
0131 03:04:26.465769 (+ 1340us) client_negotiation.cc:240] Negotiation successful
0131 03:04:26.466141 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1370,"thread_start_us":126,"threads_started":1}
I20260131 03:04:26.468904 18675 heartbeater.cc:344] Connected to a master server at 127.6.22.126:34645
I20260131 03:04:26.469264 18675 heartbeater.cc:461] Registering TS with master...
I20260131 03:04:26.470000 18675 heartbeater.cc:507] Master 127.6.22.126:34645 requested a full tablet report, sending...
I20260131 03:04:26.471269 18186 ts_manager.cc:194] Registered new tserver with Master: 7a133297d4a84b8380e9bfbac28f870b (127.6.22.67:44525)
I20260131 03:04:26.472023 18186 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.67@KRBTEST.COM'} at 127.6.22.67:53649
I20260131 03:04:26.473785 6233 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:04:26.486847 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.476367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54304 (local address 127.6.22.126:34645)
0131 03:04:26.476523 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0131 03:04:26.476528 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:04:26.477146 (+ 618us) server_negotiation.cc:408] Connection header received
0131 03:04:26.477365 (+ 219us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:04:26.477368 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:04:26.477429 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:04:26.477491 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:04:26.478768 (+ 1277us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.479588 (+ 820us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.480915 (+ 1327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.481118 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.482930 (+ 1812us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:04:26.482958 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:04:26.482962 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:04:26.483004 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:04:26.484848 (+ 1844us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:04:26.485429 (+ 581us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:04:26.485433 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:04:26.485435 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:04:26.485495 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:04:26.485811 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:04:26.485815 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:04:26.485817 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:04:26.486036 (+ 219us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:04:26.486153 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0131 03:04:26.486518 (+ 365us) server_negotiation.cc:300] Negotiation successful
0131 03:04:26.486661 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260131 03:04:26.490664 18186 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:04:26.494618 18186 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:04:26.524461 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.512577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:34899 (local address 127.0.0.1:43778)
0131 03:04:26.515635 (+ 3058us) negotiation.cc:107] Waiting for socket to connect
0131 03:04:26.515648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:04:26.515743 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:04:26.518920 (+ 3177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:04:26.518923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:04:26.518944 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:04:26.519177 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.519184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.521764 (+ 2580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.521769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:04:26.523829 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.523843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.523977 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.524060 (+ 83us) client_negotiation.cc:769] Sending connection context
0131 03:04:26.524171 (+ 111us) client_negotiation.cc:240] Negotiation successful
0131 03:04:26.524294 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2996,"thread_start_us":155,"threads_started":1}
I20260131 03:04:26.527916 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.508856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:36849 (local address 127.0.0.1:46316)
0131 03:04:26.514699 (+ 5843us) negotiation.cc:107] Waiting for socket to connect
0131 03:04:26.514752 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0131 03:04:26.514907 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:04:26.518401 (+ 3494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:04:26.518408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:04:26.518434 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:04:26.518784 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.518794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.525490 (+ 6696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.525494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:04:26.527436 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.527449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.527592 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.527617 (+ 25us) client_negotiation.cc:769] Sending connection context
0131 03:04:26.527672 (+ 55us) client_negotiation.cc:240] Negotiation successful
0131 03:04:26.527754 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5622,"thread_start_us":114,"threads_started":1}
I20260131 03:04:26.529390 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.513065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.6.22.65:34899)
0131 03:04:26.515472 (+ 2407us) server_negotiation.cc:207] Beginning negotiation
0131 03:04:26.515481 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:04:26.515841 (+ 360us) server_negotiation.cc:408] Connection header received
0131 03:04:26.515946 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:04:26.515952 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:04:26.516176 (+ 224us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:04:26.516327 (+ 151us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:04:26.519340 (+ 3013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.520543 (+ 1203us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.528203 (+ 7660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.528861 (+ 658us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.529020 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
0131 03:04:26.529109 (+ 89us) server_negotiation.cc:300] Negotiation successful
0131 03:04:26.529228 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2288,"thread_start_us":86,"threads_started":1}
I20260131 03:04:26.531306 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.509835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.6.22.66:36849)
0131 03:04:26.517674 (+ 7839us) server_negotiation.cc:207] Beginning negotiation
0131 03:04:26.517682 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:04:26.517711 (+ 29us) server_negotiation.cc:408] Connection header received
0131 03:04:26.517804 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:04:26.517812 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:04:26.518035 (+ 223us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:04:26.518198 (+ 163us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:04:26.522781 (+ 4583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.525335 (+ 2554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.530319 (+ 4984us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.530873 (+ 554us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.530995 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0131 03:04:26.531064 (+ 69us) server_negotiation.cc:300] Negotiation successful
0131 03:04:26.531154 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6711,"thread_start_us":117,"threads_started":1}
I20260131 03:04:26.532645 18321 tablet_service.cc:1505] Processing CreateTablet for tablet 6024fe2fe4744694a48573f78305c35c (DEFAULT_TABLE table=test-table [id=8c96a6bf91ed4d3c9d1433a623ae8837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:04:26.533774 18321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6024fe2fe4744694a48573f78305c35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:04:26.534380 18473 tablet_service.cc:1505] Processing CreateTablet for tablet 6024fe2fe4744694a48573f78305c35c (DEFAULT_TABLE table=test-table [id=8c96a6bf91ed4d3c9d1433a623ae8837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:04:26.535395 18473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6024fe2fe4744694a48573f78305c35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:04:26.546010 18695 tablet_bootstrap.cc:492] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3: Bootstrap starting.
I20260131 03:04:26.548729 18695 tablet_bootstrap.cc:654] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3: Neither blocks nor log segments found. Creating new log.
I20260131 03:04:26.550155 18695 log.cc:826] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3: Log is configured to *not* fsync() on all Append() calls
I20260131 03:04:26.552526 18695 tablet_bootstrap.cc:492] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3: No bootstrap required, opened a new log
I20260131 03:04:26.552810 18695 ts_tablet_manager.cc:1403] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260131 03:04:26.556034 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.511596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:44525 (local address 127.0.0.1:38208)
0131 03:04:26.515139 (+ 3543us) negotiation.cc:107] Waiting for socket to connect
0131 03:04:26.515151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:04:26.515220 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:04:26.530028 (+ 14808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:04:26.530032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:04:26.530058 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:04:26.536482 (+ 6424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.536515 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.554212 (+ 17697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.554220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:04:26.555352 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.555374 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.555619 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.555658 (+ 39us) client_negotiation.cc:769] Sending connection context
0131 03:04:26.555720 (+ 62us) client_negotiation.cc:240] Negotiation successful
0131 03:04:26.555827 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3466,"thread_start_us":121,"threads_started":1}
I20260131 03:04:26.556278 18695 raft_consensus.cc:359] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8627c4e6554c94b211d22e2ffad9d3" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36849 } } peers { permanent_uuid: "7a133297d4a84b8380e9bfbac28f870b" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 44525 } } peers { permanent_uuid: "c9716dc4bd7f4010a17153de002ba057" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 34899 } }
I20260131 03:04:26.556559 18695 raft_consensus.cc:385] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:04:26.557801 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.512219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.6.22.67:44525)
0131 03:04:26.529330 (+ 17111us) server_negotiation.cc:207] Beginning negotiation
0131 03:04:26.529338 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:04:26.529364 (+ 26us) server_negotiation.cc:408] Connection header received
0131 03:04:26.529469 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:04:26.529475 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:04:26.529704 (+ 229us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:04:26.529868 (+ 164us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:04:26.552788 (+ 22920us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.553946 (+ 1158us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.556694 (+ 2748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.557300 (+ 606us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.557442 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0131 03:04:26.557517 (+ 75us) server_negotiation.cc:300] Negotiation successful
0131 03:04:26.557629 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16965,"thread_start_us":166,"threads_started":1}
I20260131 03:04:26.556643 18695 raft_consensus.cc:740] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8627c4e6554c94b211d22e2ffad9d3, State: Initialized, Role: FOLLOWER
I20260131 03:04:26.558895 18695 consensus_queue.cc:260] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8627c4e6554c94b211d22e2ffad9d3" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36849 } } peers { permanent_uuid: "7a133297d4a84b8380e9bfbac28f870b" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 44525 } } peers { permanent_uuid: "c9716dc4bd7f4010a17153de002ba057" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 34899 } }
I20260131 03:04:26.559921 18695 ts_tablet_manager.cc:1434] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260131 03:04:26.560899 18539 heartbeater.cc:499] Master 127.6.22.126:34645 was elected leader, sending a full tablet report...
I20260131 03:04:26.561012 18609 tablet_service.cc:1505] Processing CreateTablet for tablet 6024fe2fe4744694a48573f78305c35c (DEFAULT_TABLE table=test-table [id=8c96a6bf91ed4d3c9d1433a623ae8837]), partition=RANGE (key) PARTITION UNBOUNDED
W20260131 03:04:26.561563 18540 tablet.cc:2378] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:04:26.562677 18694 tablet_bootstrap.cc:492] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057: Bootstrap starting.
I20260131 03:04:26.565217 18694 tablet_bootstrap.cc:654] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057: Neither blocks nor log segments found. Creating new log.
I20260131 03:04:26.566079 18694 log.cc:826] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057: Log is configured to *not* fsync() on all Append() calls
I20260131 03:04:26.573271 18609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6024fe2fe4744694a48573f78305c35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:04:26.585614 18694 tablet_bootstrap.cc:492] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057: No bootstrap required, opened a new log
I20260131 03:04:26.586022 18694 ts_tablet_manager.cc:1403] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20260131 03:04:26.589193 18700 tablet_bootstrap.cc:492] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b: Bootstrap starting.
I20260131 03:04:26.590533 18694 raft_consensus.cc:359] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8627c4e6554c94b211d22e2ffad9d3" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36849 } } peers { permanent_uuid: "7a133297d4a84b8380e9bfbac28f870b" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 44525 } } peers { permanent_uuid: "c9716dc4bd7f4010a17153de002ba057" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 34899 } }
I20260131 03:04:26.590893 18694 raft_consensus.cc:385] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:04:26.590981 18694 raft_consensus.cc:740] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9716dc4bd7f4010a17153de002ba057, State: Initialized, Role: FOLLOWER
I20260131 03:04:26.591629 18694 consensus_queue.cc:260] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8627c4e6554c94b211d22e2ffad9d3" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36849 } } peers { permanent_uuid: "7a133297d4a84b8380e9bfbac28f870b" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 44525 } } peers { permanent_uuid: "c9716dc4bd7f4010a17153de002ba057" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 34899 } }
I20260131 03:04:26.592159 18700 tablet_bootstrap.cc:654] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b: Neither blocks nor log segments found. Creating new log.
I20260131 03:04:26.593025 18694 ts_tablet_manager.cc:1434] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260131 03:04:26.593699 18401 heartbeater.cc:499] Master 127.6.22.126:34645 was elected leader, sending a full tablet report...
I20260131 03:04:26.595329 18700 log.cc:826] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b: Log is configured to *not* fsync() on all Append() calls
I20260131 03:04:26.597721 18700 tablet_bootstrap.cc:492] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b: No bootstrap required, opened a new log
I20260131 03:04:26.597982 18700 ts_tablet_manager.cc:1403] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260131 03:04:26.601747 18700 raft_consensus.cc:359] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8627c4e6554c94b211d22e2ffad9d3" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36849 } } peers { permanent_uuid: "7a133297d4a84b8380e9bfbac28f870b" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 44525 } } peers { permanent_uuid: "c9716dc4bd7f4010a17153de002ba057" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 34899 } }
I20260131 03:04:26.602123 18700 raft_consensus.cc:385] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:04:26.602182 18700 raft_consensus.cc:740] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a133297d4a84b8380e9bfbac28f870b, State: Initialized, Role: FOLLOWER
I20260131 03:04:26.602782 18700 consensus_queue.cc:260] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8627c4e6554c94b211d22e2ffad9d3" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36849 } } peers { permanent_uuid: "7a133297d4a84b8380e9bfbac28f870b" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 44525 } } peers { permanent_uuid: "c9716dc4bd7f4010a17153de002ba057" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 34899 } }
I20260131 03:04:26.603878 18700 ts_tablet_manager.cc:1434] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260131 03:04:26.604386 18675 heartbeater.cc:499] Master 127.6.22.126:34645 was elected leader, sending a full tablet report...
Jan 31 03:04:26 dist-test-slave-j368 krb5kdc[17683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828665, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
W20260131 03:04:26.696415 18676 tablet.cc:2378] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:04:26.726195 18402 tablet.cc:2378] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:04:26.745970 18701 raft_consensus.cc:493] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:04:26.746209 18701 raft_consensus.cc:515] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8627c4e6554c94b211d22e2ffad9d3" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36849 } } peers { permanent_uuid: "7a133297d4a84b8380e9bfbac28f870b" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 44525 } } peers { permanent_uuid: "c9716dc4bd7f4010a17153de002ba057" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 34899 } }
I20260131 03:04:26.747689 18701 leader_election.cc:290] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e8627c4e6554c94b211d22e2ffad9d3 (127.6.22.66:36849), 7a133297d4a84b8380e9bfbac28f870b (127.6.22.67:44525)
I20260131 03:04:26.760210 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.749627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:54689 (local address 127.6.22.67:44525)
0131 03:04:26.749785 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0131 03:04:26.749790 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:04:26.756043 (+ 6253us) server_negotiation.cc:408] Connection header received
0131 03:04:26.756176 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:04:26.756179 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:04:26.756259 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:04:26.756346 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:04:26.756963 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.757795 (+ 832us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.759140 (+ 1345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.759655 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.759708 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0131 03:04:26.759945 (+ 237us) server_negotiation.cc:300] Negotiation successful
0131 03:04:26.760054 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260131 03:04:26.761104 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.749210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:44525 (local address 127.6.22.65:54689)
0131 03:04:26.755922 (+ 6712us) negotiation.cc:107] Waiting for socket to connect
0131 03:04:26.755941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:04:26.756078 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:04:26.756459 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:04:26.756465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:04:26.756486 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:04:26.756813 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.756825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.757980 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.757983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:04:26.758985 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.758995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.759785 (+ 790us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.759805 (+ 20us) client_negotiation.cc:769] Sending connection context
0131 03:04:26.760861 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0131 03:04:26.760946 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6616,"thread_start_us":111,"threads_started":1}
I20260131 03:04:26.762038 18629 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6024fe2fe4744694a48573f78305c35c" candidate_uuid: "c9716dc4bd7f4010a17153de002ba057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a133297d4a84b8380e9bfbac28f870b" is_pre_election: true
I20260131 03:04:26.762369 18629 raft_consensus.cc:2468] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9716dc4bd7f4010a17153de002ba057 in term 0.
I20260131 03:04:26.763157 18291 leader_election.cc:304] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a133297d4a84b8380e9bfbac28f870b, c9716dc4bd7f4010a17153de002ba057; no voters:
I20260131 03:04:26.763504 18701 raft_consensus.cc:2804] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:04:26.763576 18701 raft_consensus.cc:493] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:04:26.763618 18701 raft_consensus.cc:3060] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:04:26.764977 18701 raft_consensus.cc:515] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8627c4e6554c94b211d22e2ffad9d3" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36849 } } peers { permanent_uuid: "7a133297d4a84b8380e9bfbac28f870b" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 44525 } } peers { permanent_uuid: "c9716dc4bd7f4010a17153de002ba057" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 34899 } }
I20260131 03:04:26.765429 18701 leader_election.cc:290] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [CANDIDATE]: Term 1 election: Requested vote from peers 1e8627c4e6554c94b211d22e2ffad9d3 (127.6.22.66:36849), 7a133297d4a84b8380e9bfbac28f870b (127.6.22.67:44525)
I20260131 03:04:26.766041 18629 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6024fe2fe4744694a48573f78305c35c" candidate_uuid: "c9716dc4bd7f4010a17153de002ba057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a133297d4a84b8380e9bfbac28f870b"
I20260131 03:04:26.766204 18629 raft_consensus.cc:3060] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:04:26.767567 18629 raft_consensus.cc:2468] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9716dc4bd7f4010a17153de002ba057 in term 1.
I20260131 03:04:26.768009 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.748800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:44677 (local address 127.6.22.66:36849)
0131 03:04:26.748951 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0131 03:04:26.748956 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:04:26.755292 (+ 6336us) server_negotiation.cc:408] Connection header received
0131 03:04:26.755545 (+ 253us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:04:26.755549 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:04:26.755607 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:04:26.755660 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:04:26.762058 (+ 6398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.762753 (+ 695us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.766777 (+ 4024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.767351 (+ 574us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.767404 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0131 03:04:26.767781 (+ 377us) server_negotiation.cc:300] Negotiation successful
0131 03:04:26.767873 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260131 03:04:26.768000 18291 leader_election.cc:304] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a133297d4a84b8380e9bfbac28f870b, c9716dc4bd7f4010a17153de002ba057; no voters:
I20260131 03:04:26.768280 18701 raft_consensus.cc:2804] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:04:26.768528 18701 raft_consensus.cc:697] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [term 1 LEADER]: Becoming Leader. State: Replica: c9716dc4bd7f4010a17153de002ba057, State: Running, Role: LEADER
I20260131 03:04:26.768743 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.748211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.66:36849 (local address 127.6.22.65:44677)
0131 03:04:26.755142 (+ 6931us) negotiation.cc:107] Waiting for socket to connect
0131 03:04:26.755165 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0131 03:04:26.755404 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:04:26.755817 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:04:26.755820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:04:26.755841 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:04:26.761883 (+ 6042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.761896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.765455 (+ 3559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.765459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:04:26.766591 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:04:26.766605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.767523 (+ 918us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.767548 (+ 25us) client_negotiation.cc:769] Sending connection context
0131 03:04:26.768384 (+ 836us) client_negotiation.cc:240] Negotiation successful
0131 03:04:26.768470 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6801,"thread_start_us":122,"threads_started":1}
I20260131 03:04:26.768959 18701 consensus_queue.cc:237] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8627c4e6554c94b211d22e2ffad9d3" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36849 } } peers { permanent_uuid: "7a133297d4a84b8380e9bfbac28f870b" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 44525 } } peers { permanent_uuid: "c9716dc4bd7f4010a17153de002ba057" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 34899 } }
I20260131 03:04:26.771806 18478 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6024fe2fe4744694a48573f78305c35c" candidate_uuid: "c9716dc4bd7f4010a17153de002ba057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8627c4e6554c94b211d22e2ffad9d3" is_pre_election: true
I20260131 03:04:26.772120 18478 raft_consensus.cc:2468] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9716dc4bd7f4010a17153de002ba057 in term 0.
I20260131 03:04:26.772536 18491 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6024fe2fe4744694a48573f78305c35c" candidate_uuid: "c9716dc4bd7f4010a17153de002ba057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8627c4e6554c94b211d22e2ffad9d3"
I20260131 03:04:26.772684 18491 raft_consensus.cc:3060] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:04:26.773901 18491 raft_consensus.cc:2468] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9716dc4bd7f4010a17153de002ba057 in term 1.
I20260131 03:04:26.776172 18186 catalog_manager.cc:5654] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9716dc4bd7f4010a17153de002ba057 (127.6.22.65). New cstate: current_term: 1 leader_uuid: "c9716dc4bd7f4010a17153de002ba057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8627c4e6554c94b211d22e2ffad9d3" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a133297d4a84b8380e9bfbac28f870b" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 44525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9716dc4bd7f4010a17153de002ba057" member_type: VOTER last_known_addr { host: "127.6.22.65" port: 34899 } health_report { overall_health: HEALTHY } } }
I20260131 03:04:26.824108 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.819320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43780 (local address 127.6.22.65:34899)
0131 03:04:26.819595 (+ 275us) server_negotiation.cc:207] Beginning negotiation
0131 03:04:26.819600 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:04:26.819611 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:04:26.819688 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:04:26.819691 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:04:26.819747 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:04:26.819862 (+ 115us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:04:26.820332 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.821119 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:04:26.822215 (+ 1096us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:04:26.822427 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:04:26.822567 (+ 140us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:04:26.823253 (+ 686us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:04:26.823393 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0131 03:04:26.823892 (+ 499us) server_negotiation.cc:300] Negotiation successful
0131 03:04:26.823955 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260131 03:04:26.832100 18629 raft_consensus.cc:1275] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b [term 1 FOLLOWER]: Refusing update from remote peer c9716dc4bd7f4010a17153de002ba057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:04:26.833348 18701 consensus_queue.cc:1048] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a133297d4a84b8380e9bfbac28f870b" member_type: VOTER last_known_addr { host: "127.6.22.67" port: 44525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:04:26.834954 18491 raft_consensus.cc:1275] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3 [term 1 FOLLOWER]: Refusing update from remote peer c9716dc4bd7f4010a17153de002ba057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:04:26.835817 18707 consensus_queue.cc:1048] T 6024fe2fe4744694a48573f78305c35c P c9716dc4bd7f4010a17153de002ba057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8627c4e6554c94b211d22e2ffad9d3" member_type: VOTER last_known_addr { host: "127.6.22.66" port: 36849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:04:26.848306 18710 mvcc.cc:204] Tried to move back new op lower bound from 7249218219331895296 to 7249218219089379328. Current Snapshot: MvccSnapshot[applied={T|T < 7249218219331895296}]
I20260131 03:04:26.850917 18711 mvcc.cc:204] Tried to move back new op lower bound from 7249218219331895296 to 7249218219089379328. Current Snapshot: MvccSnapshot[applied={T|T < 7249218219331895296}]
I20260131 03:04:26.864930 18182 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:04:26.865114 18182 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:04:26.866310 18713 mvcc.cc:204] Tried to move back new op lower bound from 7249218219331895296 to 7249218219089379328. Current Snapshot: MvccSnapshot[applied={T|T < 7249218219331895296}]
I20260131 03:04:26.868556 18182 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 689d75d8148642dfb2f24e2955b4285d: Sending DeleteTablet for 3 replicas of tablet 6024fe2fe4744694a48573f78305c35c
I20260131 03:04:26.870024 18609 tablet_service.cc:1552] Processing DeleteTablet for tablet 6024fe2fe4744694a48573f78305c35c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:04:26 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:38208
I20260131 03:04:26.870350 18473 tablet_service.cc:1552] Processing DeleteTablet for tablet 6024fe2fe4744694a48573f78305c35c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:04:26 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:46316
I20260131 03:04:26.870584 18321 tablet_service.cc:1552] Processing DeleteTablet for tablet 6024fe2fe4744694a48573f78305c35c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:04:26 UTC) from {username='kudu', principal='kudu/127.6.22.126@KRBTEST.COM'} at 127.0.0.1:43778
I20260131 03:04:26.872958 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 18268
I20260131 03:04:26.874141 18720 tablet_replica.cc:333] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3: stopping tablet replica
I20260131 03:04:26.874508 18720 raft_consensus.cc:2243] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:04:26.874912 18720 raft_consensus.cc:2272] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:04:26.876904 18719 tablet_replica.cc:333] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b: stopping tablet replica
I20260131 03:04:26.877213 18719 raft_consensus.cc:2243] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:04:26.877516 18719 raft_consensus.cc:2272] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:04:26.877907 18720 ts_tablet_manager.cc:1916] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:04:26.880947 18720 ts_tablet_manager.cc:1929] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:04:26.881058 18720 log.cc:1199] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-1/wal/wals/6024fe2fe4744694a48573f78305c35c
I20260131 03:04:26.881405 18720 ts_tablet_manager.cc:1950] T 6024fe2fe4744694a48573f78305c35c P 1e8627c4e6554c94b211d22e2ffad9d3: Deleting consensus metadata
I20260131 03:04:26.883389 18172 catalog_manager.cc:4985] TS 1e8627c4e6554c94b211d22e2ffad9d3 (127.6.22.66:36849): tablet 6024fe2fe4744694a48573f78305c35c (table test-table [id=8c96a6bf91ed4d3c9d1433a623ae8837]) successfully deleted
W20260131 03:04:26.887768 18171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.22.65:34899 (error 108)
I20260131 03:04:26.888886 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 18409
I20260131 03:04:26.888952 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.888631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.65:34899 (local address 127.0.0.1:43794)
0131 03:04:26.888795 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0131 03:04:26.888860 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.65:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260131 03:04:26.891357 18171 catalog_manager.cc:4712] TS c9716dc4bd7f4010a17153de002ba057 (127.6.22.65:34899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6024fe2fe4744694a48573f78305c35c: Network error: Client connection negotiation failed: client connection to 127.6.22.65:34899: connect: Connection refused (error 111)
I20260131 03:04:26.893416 18719 ts_tablet_manager.cc:1916] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:04:26.895843 18719 ts_tablet_manager.cc:1929] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:04:26.895977 18719 log.cc:1199] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b: Deleting WAL directory at /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828516572523-6233-0/minicluster-data/ts-2/wal/wals/6024fe2fe4744694a48573f78305c35c
I20260131 03:04:26.900745 18719 ts_tablet_manager.cc:1950] T 6024fe2fe4744694a48573f78305c35c P 7a133297d4a84b8380e9bfbac28f870b: Deleting consensus metadata
I20260131 03:04:26.901644 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 18546
I20260131 03:04:26.913869 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:04:26.913374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.67:44525 (local address 127.0.0.1:38220)
0131 03:04:26.913648 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0131 03:04:26.913730 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.67:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260131 03:04:26.914508 6233 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMlJawA/build/debug/bin/kudu with pid 18154
I20260131 03:04:26.924487 6233 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260131 03:04:27.271440 6233 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260131 03:04:27.271612 6233 mini_ranger.cc:66] Stopping Ranger...
I20260131 03:04:27.619345 6233 mini_ranger.cc:68] Stopped Ranger
2026-01-31 03:04:27.619 UTC [17714] LOG: received smart shutdown request
2026-01-31 03:04:27.623 UTC [17714] LOG: background worker "logical replication launcher" (PID 17722) exited with exit code 1
2026-01-31 03:04:27.624 UTC [17717] LOG: shutting down
2026-01-31 03:04:27.626 UTC [17717] LOG: checkpoint starting: shutdown immediate
2026-01-31 03:04:28.587 UTC [17717] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.037 s, sync=0.919 s, total=0.964 s; sync files=1108, longest=0.005 s, average=0.001 s; distance=12074 kB, estimate=12074 kB; lsn=0/2045140, redo lsn=0/2045140
2026-01-31 03:04:28.597 UTC [17714] LOG: database system is shut down
2026-01-31T03:04:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53980 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:04:28 dist-test-slave-j368 krb5kdc[18733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:04:28 dist-test-slave-j368 krb5kdc[18733](info): set up 2 sockets
Jan 31 03:04:28 dist-test-slave-j368 krb5kdc[18733](info): commencing operation
krb5kdc: starting...
W20260131 03:04:30.747301 6233 mini_kdc.cc:121] Time spent starting KDC: real 1.999s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/test-user.keytab.
Jan 31 03:04:30 dist-test-slave-j368 krb5kdc[18733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:04:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:04:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.22.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.22.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/rangeradmin_127.6.22.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/rangeradmin_127.6.22.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.22.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.22.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/rangerlookup_127.6.22.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/rangerlookup_127.6.22.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.22.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/HTTP_127.6.22.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/HTTP_127.6.22.84@KRBTEST.COM.keytab.
I20260131 03:04:30.923699 6233 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMlJawA/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/postgres -l logfile start
2026-01-31 03:04:33.618 UTC [18764] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-01-31 03:04:33.618 UTC [18764] LOG: listening on IPv4 address "127.6.22.84", port 41463
2026-01-31 03:04:33.621 UTC [18764] LOG: listening on Unix socket "/tmp/.s.PGSQL.41463"
2026-01-31 03:04:33.627 UTC [18769] LOG: database system was shut down at 2026-01-31 03:04:32 UTC
2026-01-31 03:04:33.632 UTC [18764] LOG: database system is ready to accept connections
I20260131 03:04:35.547417 6233 mini_postgres.cc:96] Postgres bound to 41463
2026-01-31 03:04:35.556 UTC [18775] FATAL: database "slave" does not exist
127.6.22.84:41463 - accepting connections
I20260131 03:04:35.558041 6233 mini_ranger.cc:161] Starting Ranger...
I20260131 03:04:35.583552 6233 mini_ranger.cc:84] Created miniranger Postgres user
I20260131 03:04:35.661593 6233 mini_ranger.cc:87] Created ranger Postgres database
I20260131 03:04:35.661679 6233 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-admin
2026-01-31 03:04:35,939 [I] DB FLAVOR :POSTGRES
2026-01-31 03:04:35,939 [I] --------- Verifying Ranger DB connection ---------
2026-01-31 03:04:35,939 [I] Checking connection..
2026-01-31 03:04:35,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-01-31 03:04:36,272 [I] Checking connection passed.
2026-01-31 03:04:36,272 [I] --------- Verifying version history table ---------
2026-01-31 03:04:36,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-31 03:04:36,599 [I] Table x_db_version_h does not exist in database ranger
2026-01-31 03:04:36,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-31 03:04:36,936 [I] Table x_db_version_h does not exist in database ranger
2026-01-31 03:04:36,936 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-01-31 03:04:36,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-01-31 03:04:37.271 UTC [18842] WARNING: there is no transaction in progress
2026-01-31 03:04:37,279 [I] create_dbversion_catalog.sql file imported successfully
2026-01-31 03:04:37,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-31 03:04:37,601 [I] Table x_db_version_h already exists in database 'ranger'
2026-01-31 03:04:37,601 [I] --------- Importing Ranger Core DB Schema ---------
2026-01-31 03:04:37,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-01-31 03:04:37,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-01-31 03:04:38,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j368.c.gcp-upstream.internal','N') ;"
2026-01-31 03:04:38,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-01-31 03:04:38,952 [I] Table x_portal_user does not exist in database ranger
2026-01-31 03:04:38,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-01-31 03:04:39,280 [I] Table x_policy_ref_group does not exist in database ranger
2026-01-31 03:04:39,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-01-31 03:04:39,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-01-31 03:04:39,928 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-01-31 03:04:39,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-01-31 03:04:40.882 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:40.893 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:40.906 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:40.917 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:40.929 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:40.944 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:40.995 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:41.000 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:41.010 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:41.021 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:41.031 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:41.042 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:41.052 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:41.057 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:41.062 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:41.067 UTC [18959] WARNING: there is no transaction in progress
2026-01-31 03:04:41,565 [I] ranger_core_db_postgres.sql file imported successfully
2026-01-31 03:04:41,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-01-31 03:04:41,871 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-01-31 03:04:41,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-01-31 03:04:42,203 [I] Table x_portal_user already exists in database 'ranger'
2026-01-31 03:04:42,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-01-31 03:04:42,537 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-01-31 03:04:42,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-01-31 03:04:42,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-01-31 03:04:43,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j368.c.gcp-upstream.internal';"
2026-01-31 03:04:43,552 [I] CORE_DB_SCHEMA import status has been updated
2026-01-31 03:04:43,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-01-31 03:04:43,893 [I] DB_PATCHES have already been applied
I20260131 03:04:43.899092 6233 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMlJawA/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260131 03:04:43.899201 6233 mini_ranger.cc:193] Using host: 127.6.22.84
I20260131 03:04:43.902431 6233 mini_ranger.cc:227] Ranger admin URL: http://127.6.22.84:43401
Jan 31, 2026 3:04:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 31, 2026 3:04:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 31, 2026 3:04:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-admin/ews/webapp
Jan 31, 2026 3:04:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jan 31, 2026 3:04:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-admin/ews/webapp] .....
Jan 31, 2026 3:04:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-admin/ews/webapp].
Jan 31, 2026 3:04:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260131 03:04:46.112175 6233 mini_ranger.cc:160] Time spent starting Ranger: real 10.554s user 0.000s sys 0.011s
Jan 31, 2026 3:05:07 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jan 31, 2026 3:05:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
Jan 31, 2026 3:05:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jan 31, 2026 3:05:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jan 31, 2026 3:05:07 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jan 31, 2026 3:05:08 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260131 03:05:09.263120 6233 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.22.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.22.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/rangerkms_127.6.22.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.22.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/rangerkms_127.6.22.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.22.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.22.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/HTTP_127.6.22.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.22.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/HTTP_127.6.22.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jan 31 03:05:09 dist-test-slave-j368 krb5kdc[18733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828709, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260131 03:05:09.345976 6233 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260131 03:05:09.356874 6233 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260131 03:05:09.464059 6233 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260131 03:05:09.464174 6233 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-kms
I20260131 03:05:09.464191 6233 mini_ranger_kms.cc:227] Using postgres at 127.6.22.84:41463
2026-01-31 03:05:09,647 [I] DB FLAVOR :POSTGRES
2026-01-31 03:05:09,647 [I] --------- Verifying Ranger DB connection ---------
2026-01-31 03:05:09,647 [I] Checking connection
2026-01-31 03:05:09,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-01-31 03:05:10,018 [I] connection success
2026-01-31 03:05:10,019 [I] --------- Verifying Ranger DB tables ---------
2026-01-31 03:05:10,019 [I] Verifying table ranger_masterkey in database rangerkms
2026-01-31 03:05:10,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-01-31 03:05:10,406 [I] Table ranger_masterkey does not exist in database rangerkms
2026-01-31 03:05:10,407 [I] --------- Importing Ranger Core DB Schema ---------
2026-01-31 03:05:10,407 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-01-31 03:05:10,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.22.84:41463/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-01-31 03:05:10,805 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260131 03:05:11.366436 6233 mini_ranger_kms.cc:310] Created kms service
I20260131 03:05:11.608539 6233 mini_ranger_kms.cc:326] Created kudu user
I20260131 03:05:11.666276 6233 mini_ranger_kms.cc:343] Created rangerkms user
Jan 31 03:05:11 dist-test-slave-j368 krb5kdc[18733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828709, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
I20260131 03:05:12.095973 6233 mini_ranger_kms.cc:384] Added ranger policy
I20260131 03:05:12.096236 6233 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMlJawA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMlJawA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260131 03:05:12.096259 6233 mini_ranger_kms.cc:240] Using host: 127.6.22.84
I20260131 03:05:12.099736 6233 mini_ranger_kms.cc:276] Ranger KMS PID: 19151
I20260131 03:05:12.099835 6233 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.22.84:52347
Jan 31, 2026 3:05:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 31, 2026 3:05:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 31, 2026 3:05:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jan 31, 2026 3:05:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-kms/ews/webapp] .....
Jan 31, 2026 3:05:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-kms/ews/webapp].
Jan 31, 2026 3:05:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260131 03:05:14.440094 6233 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.094s user 0.006s sys 0.159s
I20260131 03:05:14.440222 6233 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260131 03:05:14.440292 6233 mini_ranger_kms.cc:401] 127.6.22.84:52347/kms/v1/keys
Jan 31 03:05:14 dist-test-slave-j368 krb5kdc[18733](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1769828714, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.22.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jan 31 03:05:15 dist-test-slave-j368 krb5kdc[18733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828714, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.22.84@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
Jan 31 03:05:15 dist-test-slave-j368 krb5kdc[18733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828714, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.22.84@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
Jan 31 03:05:18 dist-test-slave-j368 krb5kdc[18733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:05:18.167389 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:41381
--webserver_interface=127.6.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.22.126:41381
--ranger_config_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.22.84:52347/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:05:18.282753 19204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:05:18.283068 19204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:05:18.283129 19204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:05:18.286801 19204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:05:18.286895 19204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:05:18.286924 19204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:05:18.286944 19204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:05:18.287022 19204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:05:18.292007 19204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.22.84:52347/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.22.126:41381
--ranger_config_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.22.126:41381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.22.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:05:18.293503 19204 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:05:18.294512 19204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:05:18.301554 19209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:05:18.301651 19204 server_base.cc:1060] running on GCE node
W20260131 03:05:18.301556 19210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:05:18.301888 19212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:05:18.302552 19204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:05:18.303645 19204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:05:18.304833 19204 hybrid_clock.cc:648] HybridClock initialized: now 1769828718304811 us; error 59 us; skew 500 ppm
Jan 31 03:05:18 dist-test-slave-j368 krb5kdc[18733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828718, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:05:18.307904 19204 init.cc:377] Logged in from keytab as kudu/127.6.22.126@KRBTEST.COM (short username kudu)
I20260131 03:05:18.309255 19204 webserver.cc:492] Webserver started at http://127.6.22.126:40967/ using document root <none> and password file <none>
I20260131 03:05:18.309861 19204 fs_manager.cc:362] Metadata directory not provided
I20260131 03:05:18.309916 19204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:05:18.310139 19204 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 31 03:05:18 dist-test-slave-j368 krb5kdc[18733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828718, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.126@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
I20260131 03:05:18.372704 19204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/data/instance:
uuid: "6256b342a6f549a3bf692250f83d27f0"
format_stamp: "Formatted at 2026-01-31 03:05:18 on dist-test-slave-j368"
server_key: "35fc849a3926358f5d9a075ce4094127"
server_key_iv: "0d1e4a8f0185898c808373383e451a77"
server_key_version: "kuduclusterkey@0"
I20260131 03:05:18.373391 19204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/wal/instance:
uuid: "6256b342a6f549a3bf692250f83d27f0"
format_stamp: "Formatted at 2026-01-31 03:05:18 on dist-test-slave-j368"
server_key: "35fc849a3926358f5d9a075ce4094127"
server_key_iv: "0d1e4a8f0185898c808373383e451a77"
server_key_version: "kuduclusterkey@0"
I20260131 03:05:18.377734 19204 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260131 03:05:18.395797 19219 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:05:18.397254 19204 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260131 03:05:18.397423 19204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/wal
uuid: "6256b342a6f549a3bf692250f83d27f0"
format_stamp: "Formatted at 2026-01-31 03:05:18 on dist-test-slave-j368"
server_key: "35fc849a3926358f5d9a075ce4094127"
server_key_iv: "0d1e4a8f0185898c808373383e451a77"
server_key_version: "kuduclusterkey@0"
I20260131 03:05:18.397554 19204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:05:18.435801 19204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:05:18.439510 19204 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:05:18.439740 19204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:05:18.449503 19271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.126:41381 every 8 connection(s)
I20260131 03:05:18.449514 19204 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.126:41381
I20260131 03:05:18.450891 19204 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/data/info.pb
I20260131 03:05:18.454494 19272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:05:18.456513 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 19204
I20260131 03:05:18.456740 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/wal/instance
I20260131 03:05:18.457062 6233 external_mini_cluster.cc:1468] Setting key 1fd6aeb0130c1fa577b02d76ce236b0d
I20260131 03:05:18.461934 19272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0: Bootstrap starting.
I20260131 03:05:18.465061 19272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0: Neither blocks nor log segments found. Creating new log.
I20260131 03:05:18.466082 19272 log.cc:826] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0: Log is configured to *not* fsync() on all Append() calls
Jan 31 03:05:18 dist-test-slave-j368 krb5kdc[18733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
I20260131 03:05:18.468933 19272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0: No bootstrap required, opened a new log
I20260131 03:05:18.472489 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:18.458896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.6.22.126:41381)
0131 03:05:18.459486 (+ 590us) server_negotiation.cc:207] Beginning negotiation
0131 03:05:18.459498 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:05:18.459542 (+ 44us) server_negotiation.cc:408] Connection header received
0131 03:05:18.460434 (+ 892us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:05:18.460490 (+ 56us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:05:18.461209 (+ 719us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:05:18.461738 (+ 529us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:05:18.462834 (+ 1096us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:05:18.463657 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:05:18.464373 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:05:18.464731 (+ 358us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:05:18.467547 (+ 2816us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:05:18.467575 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:05:18.467593 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:05:18.467635 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:05:18.469623 (+ 1988us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:05:18.470095 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:05:18.470101 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:05:18.470107 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:05:18.470179 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:05:18.470432 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:05:18.470435 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:05:18.470437 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:05:18.470897 (+ 460us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:05:18.471188 (+ 291us) server_negotiation.cc:1036] Waiting for connection context
0131 03:05:18.471405 (+ 217us) server_negotiation.cc:300] Negotiation successful
0131 03:05:18.471664 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":153,"threads_started":1}
I20260131 03:05:18.472986 19272 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6256b342a6f549a3bf692250f83d27f0" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41381 } }
I20260131 03:05:18.473281 19272 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:05:18.473420 19272 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6256b342a6f549a3bf692250f83d27f0, State: Initialized, Role: FOLLOWER
I20260131 03:05:18.473981 19272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6256b342a6f549a3bf692250f83d27f0" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41381 } }
I20260131 03:05:18.474144 19272 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:05:18.474243 19272 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:05:18.474347 19272 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:05:18.475572 19272 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6256b342a6f549a3bf692250f83d27f0" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41381 } }
I20260131 03:05:18.476019 19272 leader_election.cc:304] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6256b342a6f549a3bf692250f83d27f0; no voters:
I20260131 03:05:18.476423 19272 leader_election.cc:290] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:05:18.476739 19277 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:05:18.477103 19277 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6256b342a6f549a3bf692250f83d27f0, State: Running, Role: LEADER
I20260131 03:05:18.477525 19277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6256b342a6f549a3bf692250f83d27f0" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41381 } }
I20260131 03:05:18.477880 19272 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:05:18.479359 19277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6256b342a6f549a3bf692250f83d27f0. Latest consensus state: current_term: 1 leader_uuid: "6256b342a6f549a3bf692250f83d27f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6256b342a6f549a3bf692250f83d27f0" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41381 } } }
I20260131 03:05:18.479624 19277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [sys.catalog]: This master's current role is: LEADER
I20260131 03:05:18.479784 19278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6256b342a6f549a3bf692250f83d27f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6256b342a6f549a3bf692250f83d27f0" member_type: VOTER last_known_addr { host: "127.6.22.126" port: 41381 } } }
I20260131 03:05:18.479910 19278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0 [sys.catalog]: This master's current role is: LEADER
I20260131 03:05:18.480700 19272 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260131 03:05:18.481419 19287 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:05:18.489370 19287 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:05:18.499827 19287 catalog_manager.cc:1357] Generated new cluster ID: d52dd6c63dfe4bb581188468f2233be0
I20260131 03:05:18.499915 19287 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:05:18.531949 19287 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:05:18.533478 19287 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:05:18.543036 19287 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 6256b342a6f549a3bf692250f83d27f0: Generated new TSK 0
I20260131 03:05:18.544106 19287 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:05:19.332206 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.22.84:52347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.65:0
--local_ip_for_outbound_sockets=127.6.22.65
--webserver_interface=127.6.22.65
--webserver_port=0
--tserver_master_addrs=127.6.22.126:41381
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:05:19.456890 19318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:05:19.457187 19318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:05:19.457330 19318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:05:19.461710 19318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:05:19.461822 19318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:05:19.461942 19318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.65
I20260131 03:05:19.466805 19318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.22.84:52347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.22.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:41381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.65
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:05:19.468156 19318 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:05:19.469244 19318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:05:19.476301 19326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:05:19.476389 19328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:05:19.476302 19325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:05:19.477391 19318 server_base.cc:1060] running on GCE node
I20260131 03:05:19.477941 19318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:05:19.478752 19318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:05:19.479957 19318 hybrid_clock.cc:648] HybridClock initialized: now 1769828719479909 us; error 82 us; skew 500 ppm
Jan 31 03:05:19 dist-test-slave-j368 krb5kdc[18733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:05:19.484072 19318 init.cc:377] Logged in from keytab as kudu/127.6.22.65@KRBTEST.COM (short username kudu)
I20260131 03:05:19.485695 19318 webserver.cc:492] Webserver started at http://127.6.22.65:37239/ using document root <none> and password file <none>
I20260131 03:05:19.486449 19318 fs_manager.cc:362] Metadata directory not provided
I20260131 03:05:19.486527 19318 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:05:19.486763 19318 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 31 03:05:19 dist-test-slave-j368 krb5kdc[18733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
I20260131 03:05:19.505002 19318 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/data/instance:
uuid: "8e473260da144122b6946ccb5b78cb25"
format_stamp: "Formatted at 2026-01-31 03:05:19 on dist-test-slave-j368"
server_key: "fc6ba4a648e6b09ae9fd21a95540495d"
server_key_iv: "9933fbc15f9936441b060989c01de68b"
server_key_version: "kuduclusterkey@0"
I20260131 03:05:19.506215 19318 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance:
uuid: "8e473260da144122b6946ccb5b78cb25"
format_stamp: "Formatted at 2026-01-31 03:05:19 on dist-test-slave-j368"
server_key: "fc6ba4a648e6b09ae9fd21a95540495d"
server_key_iv: "9933fbc15f9936441b060989c01de68b"
server_key_version: "kuduclusterkey@0"
I20260131 03:05:19.520392 19318 fs_manager.cc:696] Time spent creating directory manager: real 0.014s user 0.004s sys 0.000s
I20260131 03:05:19.549332 19335 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:05:19.550925 19318 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260131 03:05:19.551091 19318 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/wal
uuid: "8e473260da144122b6946ccb5b78cb25"
format_stamp: "Formatted at 2026-01-31 03:05:19 on dist-test-slave-j368"
server_key: "fc6ba4a648e6b09ae9fd21a95540495d"
server_key_iv: "9933fbc15f9936441b060989c01de68b"
server_key_version: "kuduclusterkey@0"
I20260131 03:05:19.551218 19318 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:05:19.580027 19318 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:05:19.583797 19318 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:05:19.584080 19318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:05:19.584864 19318 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:05:19.586091 19318 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:05:19.586199 19318 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:05:19.586329 19318 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:05:19.586412 19318 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:05:19.598016 19318 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.65:35027
I20260131 03:05:19.598429 19450 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.65:35027 every 8 connection(s)
I20260131 03:05:19.599450 19318 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/data/info.pb
I20260131 03:05:19.603787 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 19318
I20260131 03:05:19.603966 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-0/wal/instance
I20260131 03:05:19.604302 6233 external_mini_cluster.cc:1468] Setting key d6418e8c62cc9ab0c3d70b837f6a6377
WARNING: no policy specified for kudu/127.6.22.66@KRBTEST.COM; defaulting to no policy
Jan 31 03:05:19 dist-test-slave-j368 krb5kdc[18733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.65@KRBTEST.COM for kudu/127.6.22.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.22.66@KRBTEST.COM" created.
I20260131 03:05:19.626014 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:19.602120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.22.65:51477 (local address 127.6.22.126:41381)
0131 03:05:19.602414 (+ 294us) server_negotiation.cc:207] Beginning negotiation
0131 03:05:19.602420 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:05:19.610138 (+ 7718us) server_negotiation.cc:408] Connection header received
0131 03:05:19.611644 (+ 1506us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:05:19.611650 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:05:19.611742 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:05:19.611879 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:05:19.613935 (+ 2056us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:05:19.614843 (+ 908us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:05:19.615882 (+ 1039us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:05:19.616098 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:05:19.620833 (+ 4735us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:05:19.620875 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:05:19.620879 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:05:19.620921 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:05:19.622941 (+ 2020us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:05:19.623714 (+ 773us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:05:19.623719 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:05:19.623722 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:05:19.623808 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:05:19.624359 (+ 551us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:05:19.624364 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:05:19.624366 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:05:19.624594 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:05:19.624754 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0131 03:05:19.625607 (+ 853us) server_negotiation.cc:300] Negotiation successful
0131 03:05:19.625780 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
I20260131 03:05:19.627940 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:19.602487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.65:51477)
0131 03:05:19.609928 (+ 7441us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:19.609999 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0131 03:05:19.611308 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:05:19.612293 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:05:19.612308 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:05:19.612795 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:05:19.613682 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:05:19.613703 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:05:19.615049 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:05:19.615053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:05:19.615704 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:05:19.615715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:05:19.616423 (+ 708us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:05:19.617248 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:05:19.617296 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:05:19.619792 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:05:19.623145 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:05:19.623151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:05:19.623168 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:05:19.623541 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:05:19.623967 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:05:19.623971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:05:19.623974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:05:19.624204 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:05:19.624854 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:05:19.624864 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:05:19.625249 (+ 385us) client_negotiation.cc:769] Sending connection context
0131 03:05:19.626589 (+ 1340us) client_negotiation.cc:240] Negotiation successful
0131 03:05:19.626951 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7098,"thread_start_us":132,"threads_started":1}
I20260131 03:05:19.629884 19451 heartbeater.cc:344] Connected to a master server at 127.6.22.126:41381
I20260131 03:05:19.630169 19451 heartbeater.cc:461] Registering TS with master...
I20260131 03:05:19.630892 19451 heartbeater.cc:507] Master 127.6.22.126:41381 requested a full tablet report, sending...
I20260131 03:05:19.632928 19236 ts_manager.cc:194] Registered new tserver with Master: 8e473260da144122b6946ccb5b78cb25 (127.6.22.65:35027)
I20260131 03:05:19.634635 19236 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.22.65@KRBTEST.COM'} at 127.6.22.65:51477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.6.22.126 from keytab /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
WARNING: no policy specified for HTTP/127.6.22.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.22.66@KRBTEST.COM" created.
I20260131 03:05:19.662343 19310 server.cc:273] Received an EOF from the subprocess
F20260131 03:05:19.663105 19296 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f26ccdb3dcd google::LogMessage::Fail() at ??:0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.22.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab.
I20260131 03:05:19.667542 6233 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMlJawA/build/debug/bin/kudu
/tmp/dist-test-taskMlJawA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.22.84:52347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.22.66:0
--local_ip_for_outbound_sockets=127.6.22.66
--webserver_interface=127.6.22.66
--webserver_port=0
--tserver_master_addrs=127.6.22.126:41381
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.22.84:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
@ 0x7f26ccdb7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f26ccdb37cc google::LogMessage::Flush() at ??:0
@ 0x7f26ccdb4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f26cea6b2d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f26cea680d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f26cea6c005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55aba3112a0a std::function<>::operator()() at ??:0
@ 0x7f26ce20db04 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f26ceaa76db start_thread at ??:0
@ 0x7f26cbf6a71f clone at ??:0
W20260131 03:05:19.806053 19461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:05:19.806349 19461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:05:19.806438 19461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:05:19.810058 19461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:05:19.810165 19461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:05:19.810279 19461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.22.66
I20260131 03:05:19.814952 19461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.22.84:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.22.84:52347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.22.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.22.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.22.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.22.126:41381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.22.66
--log_dir=/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 31 Jan 2026 02:43:14 UTC on 24a791456cd2
build id 10270
I20260131 03:05:19.816900 19461 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:05:19.818163 19461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:05:19.827168 19476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:05:19.827194 19479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:05:19.827194 19477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:05:19.827912 19461 server_base.cc:1060] running on GCE node
I20260131 03:05:19.828516 19461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:05:19.829380 19461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:05:19.830576 19461 hybrid_clock.cc:648] HybridClock initialized: now 1769828719830547 us; error 52 us; skew 500 ppm
Jan 31 03:05:19 dist-test-slave-j368 krb5kdc[18733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:05:19.834286 19461 init.cc:377] Logged in from keytab as kudu/127.6.22.66@KRBTEST.COM (short username kudu)
I20260131 03:05:19.835799 19461 webserver.cc:492] Webserver started at http://127.6.22.66:36633/ using document root <none> and password file <none>
I20260131 03:05:19.836557 19461 fs_manager.cc:362] Metadata directory not provided
I20260131 03:05:19.836638 19461 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:05:19.836915 19461 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 31 03:05:19 dist-test-slave-j368 krb5kdc[18733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.22.66@KRBTEST.COM for HTTP/127.6.22.84@KRBTEST.COM
I20260131 03:05:19.850660 19461 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/data/instance:
uuid: "e28edb5ec8104d2fa54381ee7c340450"
format_stamp: "Formatted at 2026-01-31 03:05:19 on dist-test-slave-j368"
server_key: "ba47f7757fb18d3a36dc446353ccd67d"
server_key_iv: "c23ce1244e95966cf218f59bdf19a3b3"
server_key_version: "kuduclusterkey@0"
I20260131 03:05:19.851344 19461 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance:
uuid: "e28edb5ec8104d2fa54381ee7c340450"
format_stamp: "Formatted at 2026-01-31 03:05:19 on dist-test-slave-j368"
server_key: "ba47f7757fb18d3a36dc446353ccd67d"
server_key_iv: "c23ce1244e95966cf218f59bdf19a3b3"
server_key_version: "kuduclusterkey@0"
I20260131 03:05:19.855413 19461 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260131 03:05:19.871867 19486 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:05:19.873390 19461 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260131 03:05:19.873548 19461 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/wal
uuid: "e28edb5ec8104d2fa54381ee7c340450"
format_stamp: "Formatted at 2026-01-31 03:05:19 on dist-test-slave-j368"
server_key: "ba47f7757fb18d3a36dc446353ccd67d"
server_key_iv: "c23ce1244e95966cf218f59bdf19a3b3"
server_key_version: "kuduclusterkey@0"
I20260131 03:05:19.873693 19461 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:05:19.894522 19461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:05:19.899207 19461 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:05:19.899505 19461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:05:19.900437 19461 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:05:19.901965 19461 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:05:19.902057 19461 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:05:19.902138 19461 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:05:19.902184 19461 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:05:19.913182 19461 rpc_server.cc:307] RPC server started. Bound to: 127.6.22.66:32815
I20260131 03:05:19.913502 19599 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.22.66:32815 every 8 connection(s)
I20260131 03:05:19.914288 19461 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/data/info.pb
I20260131 03:05:19.915058 6233 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMlJawA/build/debug/bin/kudu as pid 19461
I20260131 03:05:19.915210 6233 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMlJawA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828516572523-6233-0/minicluster-data/ts-1/wal/instance
I20260131 03:05:19.915447 6233 external_mini_cluster.cc:1468] Setting key 906ddd5f559ba7101cf66e4979e6fc57
W20260131 03:05:19.923031 18731 connection.cc:537] client connection to 127.6.22.126:41381 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260131 03:05:19.923983 19602 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:05:19.916838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.66:39839)
0131 03:05:19.917383 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:19.917449 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0131 03:05:19.918209 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:05:19.923268 (+ 5059us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":140,"threads_started":1}
W20260131 03:05:19.924422 19600 heartbeater.cc:646] Failed to heartbeat to 127.6.22.126:41381 (0 consecutive failures): Network error: Failed to ping master at 127.6.22.126:41381: Client connection negotiation failed: client connection to 127.6.22.126:41381: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260131 03:05:19.925170 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:19.924920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.66:38759)
0131 03:05:19.925048 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:19.925113 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260131 03:05:19.925799 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:19.925571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.66:50925)
0131 03:05:19.925694 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:19.925740 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260131 03:05:19.925968 19600 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260131 03:05:20.638307 19451 heartbeater.cc:499] Master 127.6.22.126:41381 was elected leader, sending a full tablet report...
I20260131 03:05:20.639365 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:20.638875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.65:52101)
0131 03:05:20.639173 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:20.639249 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
W20260131 03:05:20.639770 19451 heartbeater.cc:646] Failed to heartbeat to 127.6.22.126:41381 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
I20260131 03:05:20.927227 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:20.926673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.66:37471)
0131 03:05:20.927027 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:20.927092 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":88,"threads_started":1}
I20260131 03:05:21.641400 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:21.640787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.65:47741)
0131 03:05:21.641185 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:21.641275 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":99,"threads_started":1}
I20260131 03:05:21.929126 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:21.928335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.66:51729)
0131 03:05:21.928909 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:21.928996 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":110,"threads_started":1}
I20260131 03:05:22.642963 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:22.642391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.65:35569)
0131 03:05:22.642785 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:22.642849 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":112,"threads_started":1}
I20260131 03:05:22.930760 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:22.930058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.66:55321)
0131 03:05:22.930505 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:22.930624 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":88,"threads_started":1}
I20260131 03:05:23.644438 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:23.643955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.65:35851)
0131 03:05:23.644271 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:23.644333 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":108,"threads_started":1}
I20260131 03:05:23.932600 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:23.932000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.66:35761)
0131 03:05:23.932396 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:23.932480 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":112,"threads_started":1}
I20260131 03:05:24.646003 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:24.645499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.65:51083)
0131 03:05:24.645837 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:24.645901 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":95,"threads_started":1}
I20260131 03:05:24.934569 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:05:24.934073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.22.126:41381 (local address 127.6.22.66:43155)
0131 03:05:24.934413 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0131 03:05:24.934470 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.22.126:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":140,"threads_started":1}
F20260131 03:05:24.978076 6233 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1769828724 (unix time) try "date -d @1769828724" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800001859) received by PID 6233 (TID 0x7f9f10929d80) from PID 6233; stack trace: ***
@ 0x7f9f1c5d0980 (unknown) at ??:0
@ 0x7f9f19c06fb7 gsignal at ??:0
@ 0x7f9f19c08921 abort at ??:0
@ 0x7f9f1ae74dcd google::LogMessage::Fail() at ??:0
@ 0x7f9f1ae78b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9f1ae747cc google::LogMessage::Flush() at ??:0
@ 0x7f9f1ae75f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9f1dc01cab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f9f1dbfe9c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x55d2488a992f kudu::SecurityITest::StartCluster() at ??:0
@ 0x55d248891c15 kudu::SecurityITest_TestEncryptionWithKMSIntegrationMultipleServers_Test::TestBody() at ??:0
@ 0x7f9f1be850ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f9f1be79bdb testing::Test::Run() at ??:0
@ 0x7f9f1be79d9d testing::TestInfo::Run() at ??:0
@ 0x7f9f1be7a377 testing::TestSuite::Run() at ??:0
@ 0x7f9f1be7a77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f9f1be8560d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f9f1be79e63 testing::UnitTest::Run() at ??:0
@ 0x55d2488e1c7e RUN_ALL_TESTS() at ??:0
@ 0x55d2488e0e6b main at ??:0
@ 0x7f9f19be9bf7 __libc_start_main at ??:0
@ 0x55d24887906a _start at ??:0
2026-01-31 03:05:25.204 UTC [19078] FATAL: terminating connection due to unexpected postmaster exit
2026-01-31 03:05:25.204 UTC [19182] FATAL: terminating connection due to unexpected postmaster exit
2026-01-31 03:05:25.206 UTC [19075] FATAL: terminating connection due to unexpected postmaster exit
2026-01-31 03:05:25.206 UTC [19074] FATAL: terminating connection due to unexpected postmaster exit
2026-01-31 03:05:25.208 UTC [19076] FATAL: terminating connection due to unexpected postmaster exit
2026-01-31 03:05:25.210 UTC [19079] FATAL: terminating connection due to unexpected postmaster exit