[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:02:08 dist-test-slave-gf59 krb5kdc[19254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:02:08 dist-test-slave-gf59 krb5kdc[19254](info): set up 2 sockets
Feb 01 03:02:08 dist-test-slave-gf59 krb5kdc[19254](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260201 03:02:10.005012 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.833s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:02:10 dist-test-slave-gf59 krb5kdc[19254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:02:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:10.140995 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:34275
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:34275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:10.251055 19270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:10.251319 19270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:10.251384 19270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:10.254580 19270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:02:10.254639 19270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:10.254675 19270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:10.254707 19270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:02:10.254738 19270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:02:10.258901 19270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:34275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:34275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:10.259931 19270 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:10.260752 19270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:10.266506 19278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:10.266705 19270 server_base.cc:1060] running on GCE node
W20260201 03:02:10.266645 19275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:10.266659 19276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:10.267354 19270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:10.268343 19270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:10.269515 19270 hybrid_clock.cc:648] HybridClock initialized: now 1769914930269501 us; error 30 us; skew 500 ppm
Feb 01 03:02:10 dist-test-slave-gf59 krb5kdc[19254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914930, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:10.272154 19270 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:02:10.273257 19270 webserver.cc:492] Webserver started at http://127.18.203.254:41093/ using document root <none> and password file <none>
I20260201 03:02:10.273816 19270 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:10.273875 19270 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:10.274076 19270 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:10.275897 19270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "1d00e37aa28c460b869ee54051655f46"
format_stamp: "Formatted at 2026-02-01 03:02:10 on dist-test-slave-gf59"
server_key: "d0b8ac1e11198d40d8980693bf8de034"
server_key_iv: "3e18e2dad8a0924338bf0cb1b1f3729d"
server_key_version: "encryptionkey@0"
I20260201 03:02:10.276388 19270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "1d00e37aa28c460b869ee54051655f46"
format_stamp: "Formatted at 2026-02-01 03:02:10 on dist-test-slave-gf59"
server_key: "d0b8ac1e11198d40d8980693bf8de034"
server_key_iv: "3e18e2dad8a0924338bf0cb1b1f3729d"
server_key_version: "encryptionkey@0"
I20260201 03:02:10.279873 19270 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260201 03:02:10.281880 19285 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:10.282842 19270 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:10.282943 19270 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "1d00e37aa28c460b869ee54051655f46"
format_stamp: "Formatted at 2026-02-01 03:02:10 on dist-test-slave-gf59"
server_key: "d0b8ac1e11198d40d8980693bf8de034"
server_key_iv: "3e18e2dad8a0924338bf0cb1b1f3729d"
server_key_version: "encryptionkey@0"
I20260201 03:02:10.283028 19270 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:10.309368 19270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:10.311971 19270 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:10.312148 19270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:10.319183 19337 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:34275 every 8 connection(s)
I20260201 03:02:10.319185 19270 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:34275
I20260201 03:02:10.320240 19270 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:02:10.322868 19338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:10.328203 19338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46: Bootstrap starting.
I20260201 03:02:10.328235 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 19270
I20260201 03:02:10.328354 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:02:10.329308 19247 external_mini_cluster.cc:1468] Setting key fa9286343b33a76af2b22cb995a7ca1e
I20260201 03:02:10.330413 19338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:10.331212 19338 log.cc:826] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:10.333354 19338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46: No bootstrap required, opened a new log
I20260201 03:02:10.336614 19338 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d00e37aa28c460b869ee54051655f46" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34275 } }
I20260201 03:02:10.336830 19338 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:10.336891 19338 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00e37aa28c460b869ee54051655f46, State: Initialized, Role: FOLLOWER
I20260201 03:02:10.337374 19338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d00e37aa28c460b869ee54051655f46" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34275 } }
I20260201 03:02:10.337522 19338 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:02:10.337585 19338 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:02:10.337673 19338 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:10.338580 19338 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d00e37aa28c460b869ee54051655f46" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34275 } }
I20260201 03:02:10.338909 19338 leader_election.cc:304] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d00e37aa28c460b869ee54051655f46; no voters:
I20260201 03:02:10.339183 19338 leader_election.cc:290] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:02:10.339291 19343 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:10.339475 19343 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [term 1 LEADER]: Becoming Leader. State: Replica: 1d00e37aa28c460b869ee54051655f46, State: Running, Role: LEADER
I20260201 03:02:10.339874 19343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d00e37aa28c460b869ee54051655f46" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34275 } }
I20260201 03:02:10.340333 19338 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:02:10.341611 19345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d00e37aa28c460b869ee54051655f46. Latest consensus state: current_term: 1 leader_uuid: "1d00e37aa28c460b869ee54051655f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d00e37aa28c460b869ee54051655f46" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34275 } } }
I20260201 03:02:10.341612 19344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d00e37aa28c460b869ee54051655f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d00e37aa28c460b869ee54051655f46" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34275 } } }
I20260201 03:02:10.341742 19344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:10.341742 19345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46 [sys.catalog]: This master's current role is: LEADER
Feb 01 03:02:10 dist-test-slave-gf59 krb5kdc[19254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
W20260201 03:02:10.345281 19358 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260201 03:02:10.345386 19358 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260201 03:02:10.345702 19352 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:02:10.346596 19352 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:02:10.349174 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:10.331366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.18.203.254:34275)
0201 03:02:10.331777 (+ 411us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:10.331786 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:10.332490 (+ 704us) server_negotiation.cc:408] Connection header received
0201 03:02:10.335394 (+ 2904us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:10.335411 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:10.335715 (+ 304us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:10.335982 (+ 267us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:10.338189 (+ 2207us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:10.338977 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:10.339820 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:10.340094 (+ 274us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:10.344497 (+ 4403us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:10.344520 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:10.344532 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:10.344575 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:10.346406 (+ 1831us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:10.346934 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:10.346939 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:10.346943 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:10.347016 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:10.347285 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:10.347289 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:10.347291 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:10.347633 (+ 342us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:10.347843 (+ 210us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:10.348311 (+ 468us) server_negotiation.cc:300] Negotiation successful
0201 03:02:10.348537 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":114,"threads_started":1}
I20260201 03:02:10.351789 19352 catalog_manager.cc:1357] Generated new cluster ID: e6499ed515c84c4abc05365cfe383b40
I20260201 03:02:10.351847 19352 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:02:10.371696 19352 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:02:10.372742 19352 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:02:10.382630 19352 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 1d00e37aa28c460b869ee54051655f46: Generated new TSK 0
I20260201 03:02:10.383510 19352 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:10.432857 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:34275
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:10.544318 19366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:10.544589 19366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:10.544661 19366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:10.548007 19366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:10.548084 19366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:10.548188 19366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:02:10.552434 19366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:34275
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:10.553503 19366 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:10.554373 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:10.561036 19371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:10.561120 19372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:10.561337 19366 server_base.cc:1060] running on GCE node
W20260201 03:02:10.561053 19374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:10.561759 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:10.562314 19366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:10.563475 19366 hybrid_clock.cc:648] HybridClock initialized: now 1769914930563454 us; error 39 us; skew 500 ppm
Feb 01 03:02:10 dist-test-slave-gf59 krb5kdc[19254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914930, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:10.566418 19366 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:02:10.567525 19366 webserver.cc:492] Webserver started at http://127.18.203.193:38925/ using document root <none> and password file <none>
I20260201 03:02:10.568125 19366 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:10.568178 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:10.568370 19366 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:10.570106 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "7ef585e8ccd94009bf6aa420b534245e"
format_stamp: "Formatted at 2026-02-01 03:02:10 on dist-test-slave-gf59"
server_key: "c8039a0d3be9f67e2b6349f98cf746e9"
server_key_iv: "9147cd28a93142f283c3689d255342c3"
server_key_version: "encryptionkey@0"
I20260201 03:02:10.570564 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "7ef585e8ccd94009bf6aa420b534245e"
format_stamp: "Formatted at 2026-02-01 03:02:10 on dist-test-slave-gf59"
server_key: "c8039a0d3be9f67e2b6349f98cf746e9"
server_key_iv: "9147cd28a93142f283c3689d255342c3"
server_key_version: "encryptionkey@0"
I20260201 03:02:10.574121 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:02:10.576419 19381 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:10.577440 19366 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:02:10.577533 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "7ef585e8ccd94009bf6aa420b534245e"
format_stamp: "Formatted at 2026-02-01 03:02:10 on dist-test-slave-gf59"
server_key: "c8039a0d3be9f67e2b6349f98cf746e9"
server_key_iv: "9147cd28a93142f283c3689d255342c3"
server_key_version: "encryptionkey@0"
I20260201 03:02:10.577611 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:10.601938 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:10.604969 19366 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:10.605131 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:10.605648 19366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:10.606557 19366 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:10.606608 19366 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:10.606648 19366 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:10.606669 19366 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:10.615262 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:35897
I20260201 03:02:10.615293 19494 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:35897 every 8 connection(s)
I20260201 03:02:10.616297 19366 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:02:10.619858 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 19366
I20260201 03:02:10.619973 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:02:10.620218 19247 external_mini_cluster.cc:1468] Setting key e229b02711c3dc54014963d3a6dd6cc3
Feb 01 03:02:10 dist-test-slave-gf59 krb5kdc[19254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914930, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:10.628454 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:10.617999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:57313 (local address 127.18.203.254:34275)
0201 03:02:10.618125 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:10.618129 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:10.618822 (+ 693us) server_negotiation.cc:408] Connection header received
0201 03:02:10.619689 (+ 867us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:10.619693 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:10.619757 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:10.619865 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:10.621699 (+ 1834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:10.622203 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:10.622845 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:10.623032 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:10.625315 (+ 2283us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:10.625331 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:10.625334 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:10.625359 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:10.626642 (+ 1283us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:10.627095 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:10.627098 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:10.627100 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:10.627143 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:10.627392 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:10.627396 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:10.627398 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:10.627547 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:10.627664 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:10.628155 (+ 491us) server_negotiation.cc:300] Negotiation successful
0201 03:02:10.628284 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260201 03:02:10.629169 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:10.618243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:34275 (local address 127.18.203.193:57313)
0201 03:02:10.618664 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:10.618702 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:10.619453 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:10.620032 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:10.620045 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:10.620475 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:10.621556 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:10.621571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:10.622319 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:10.622322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:10.622739 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:10.622746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:10.622959 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:10.623537 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:10.623564 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:10.625170 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:10.626755 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:10.626760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:10.626773 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:10.626998 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:10.627220 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:10.627223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:10.627225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:10.627313 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:10.627665 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:10.627672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:10.627913 (+ 241us) client_negotiation.cc:769] Sending connection context
0201 03:02:10.628111 (+ 198us) client_negotiation.cc:240] Negotiation successful
0201 03:02:10.628380 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
I20260201 03:02:10.630323 19495 heartbeater.cc:344] Connected to a master server at 127.18.203.254:34275
I20260201 03:02:10.630565 19495 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:10.631093 19495 heartbeater.cc:507] Master 127.18.203.254:34275 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
I20260201 03:02:10.632694 19302 ts_manager.cc:194] Registered new tserver with Master: 7ef585e8ccd94009bf6aa420b534245e (127.18.203.193:35897)
I20260201 03:02:10.634089 19302 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:57313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:10.666181 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:34275
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:10.776615 19502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:10.776871 19502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:10.776953 19502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:10.780175 19502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:10.780239 19502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:10.780339 19502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:02:10.784590 19502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:34275
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:10.785669 19502 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:10.786517 19502 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:10.793045 19508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:10.793016 19507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:10.793031 19510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:10.793221 19502 server_base.cc:1060] running on GCE node
I20260201 03:02:10.793625 19502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:10.794155 19502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:10.795308 19502 hybrid_clock.cc:648] HybridClock initialized: now 1769914930795297 us; error 33 us; skew 500 ppm
Feb 01 03:02:10 dist-test-slave-gf59 krb5kdc[19254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914930, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:10.797892 19502 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:02:10.799144 19502 webserver.cc:492] Webserver started at http://127.18.203.194:42685/ using document root <none> and password file <none>
I20260201 03:02:10.799746 19502 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:10.799803 19502 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:10.799985 19502 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:10.801843 19502 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "aeab9e6df58943c0ab52a191a472d474"
format_stamp: "Formatted at 2026-02-01 03:02:10 on dist-test-slave-gf59"
server_key: "9f0f93a00b4509bc5d5d46a383495654"
server_key_iv: "53df1a8878743f3d49dad3a59fc0da74"
server_key_version: "encryptionkey@0"
I20260201 03:02:10.802397 19502 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "aeab9e6df58943c0ab52a191a472d474"
format_stamp: "Formatted at 2026-02-01 03:02:10 on dist-test-slave-gf59"
server_key: "9f0f93a00b4509bc5d5d46a383495654"
server_key_iv: "53df1a8878743f3d49dad3a59fc0da74"
server_key_version: "encryptionkey@0"
I20260201 03:02:10.806202 19502 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260201 03:02:10.808563 19517 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:10.809620 19502 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:02:10.809768 19502 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "aeab9e6df58943c0ab52a191a472d474"
format_stamp: "Formatted at 2026-02-01 03:02:10 on dist-test-slave-gf59"
server_key: "9f0f93a00b4509bc5d5d46a383495654"
server_key_iv: "53df1a8878743f3d49dad3a59fc0da74"
server_key_version: "encryptionkey@0"
I20260201 03:02:10.809870 19502 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:10.821560 19502 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:10.824508 19502 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:10.824700 19502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:10.825269 19502 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:10.826252 19502 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:10.826316 19502 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:10.826372 19502 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:10.826402 19502 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:10.834987 19502 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:35465
I20260201 03:02:10.835017 19630 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:35465 every 8 connection(s)
I20260201 03:02:10.835976 19502 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:02:10.841967 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 19502
I20260201 03:02:10.842070 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:02:10.842283 19247 external_mini_cluster.cc:1468] Setting key b525b98a216f239677776c89a9637c7e
Feb 01 03:02:10 dist-test-slave-gf59 krb5kdc[19254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914930, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:10.848277 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:10.837643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:56329 (local address 127.18.203.254:34275)
0201 03:02:10.837790 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:10.837794 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:10.838461 (+ 667us) server_negotiation.cc:408] Connection header received
0201 03:02:10.839300 (+ 839us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:10.839303 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:10.839354 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:10.839429 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:10.841118 (+ 1689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:10.841659 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:10.842456 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:10.842692 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:10.844998 (+ 2306us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:10.845016 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:10.845018 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:10.845042 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:10.846455 (+ 1413us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:10.846907 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:10.846911 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:10.846913 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:10.846959 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:10.847221 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:10.847224 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:10.847226 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:10.847377 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:10.847460 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:10.847988 (+ 528us) server_negotiation.cc:300] Negotiation successful
0201 03:02:10.848112 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260201 03:02:10.848969 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:10.837890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:34275 (local address 127.18.203.194:56329)
0201 03:02:10.838312 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:10.838349 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:10.839087 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:10.839576 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:10.839584 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:10.840018 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:10.840958 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:10.840974 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:10.841849 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:10.841853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:10.842339 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:10.842349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:10.842526 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:10.843326 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:10.843354 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:10.844852 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:10.846566 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:10.846572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:10.846585 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:10.846813 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:10.847045 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:10.847048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:10.847049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:10.847138 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:10.847473 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:10.847480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:10.847751 (+ 271us) client_negotiation.cc:769] Sending connection context
0201 03:02:10.847946 (+ 195us) client_negotiation.cc:240] Negotiation successful
0201 03:02:10.848229 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":96,"threads_started":1}
I20260201 03:02:10.850194 19631 heartbeater.cc:344] Connected to a master server at 127.18.203.254:34275
I20260201 03:02:10.850471 19631 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:10.851125 19631 heartbeater.cc:507] Master 127.18.203.254:34275 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
I20260201 03:02:10.852241 19302 ts_manager.cc:194] Registered new tserver with Master: aeab9e6df58943c0ab52a191a472d474 (127.18.203.194:35465)
I20260201 03:02:10.852820 19302 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:56329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:10.887401 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:34275
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:10.996934 19638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:10.997181 19638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:10.997254 19638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:11.000628 19638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:11.000712 19638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:11.000835 19638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:02:11.005121 19638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:34275
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:11.006258 19638 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:11.007129 19638 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:11.013984 19644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:11.014099 19638 server_base.cc:1060] running on GCE node
W20260201 03:02:11.013955 19643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:11.013984 19646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:11.014549 19638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:11.015096 19638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:11.016237 19638 hybrid_clock.cc:648] HybridClock initialized: now 1769914931016224 us; error 33 us; skew 500 ppm
Feb 01 03:02:11 dist-test-slave-gf59 krb5kdc[19254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914931, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:11.018771 19638 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:02:11.019901 19638 webserver.cc:492] Webserver started at http://127.18.203.195:36327/ using document root <none> and password file <none>
I20260201 03:02:11.020438 19638 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:11.020490 19638 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:11.020666 19638 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:11.022414 19638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "087cd8536b7842fd800ec6052e484b49"
format_stamp: "Formatted at 2026-02-01 03:02:11 on dist-test-slave-gf59"
server_key: "a0342f277606dbb2a40dd89474716841"
server_key_iv: "242b916f137162ba568cc83132a2abf9"
server_key_version: "encryptionkey@0"
I20260201 03:02:11.022922 19638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "087cd8536b7842fd800ec6052e484b49"
format_stamp: "Formatted at 2026-02-01 03:02:11 on dist-test-slave-gf59"
server_key: "a0342f277606dbb2a40dd89474716841"
server_key_iv: "242b916f137162ba568cc83132a2abf9"
server_key_version: "encryptionkey@0"
I20260201 03:02:11.026326 19638 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260201 03:02:11.028576 19653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:11.029580 19638 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:02:11.029688 19638 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "087cd8536b7842fd800ec6052e484b49"
format_stamp: "Formatted at 2026-02-01 03:02:11 on dist-test-slave-gf59"
server_key: "a0342f277606dbb2a40dd89474716841"
server_key_iv: "242b916f137162ba568cc83132a2abf9"
server_key_version: "encryptionkey@0"
I20260201 03:02:11.029778 19638 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:11.042609 19638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:11.045549 19638 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:11.045738 19638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:11.046303 19638 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:11.047209 19638 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:11.047259 19638 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:11.047312 19638 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:11.047340 19638 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:11.056205 19638 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:46595
I20260201 03:02:11.056226 19766 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:46595 every 8 connection(s)
I20260201 03:02:11.057174 19638 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:02:11.063078 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 19638
I20260201 03:02:11.063189 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:02:11.063402 19247 external_mini_cluster.cc:1468] Setting key 8a1e050d5c2cf1988e27f2be5e5b426b
Feb 01 03:02:11 dist-test-slave-gf59 krb5kdc[19254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914931, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:11.069248 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:11.058824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:47773 (local address 127.18.203.254:34275)
0201 03:02:11.058965 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:11.058969 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:11.059646 (+ 677us) server_negotiation.cc:408] Connection header received
0201 03:02:11.060505 (+ 859us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:11.060508 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:11.060566 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:11.060640 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:11.062315 (+ 1675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:11.062848 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:11.063562 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:11.063808 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:11.066146 (+ 2338us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:11.066160 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:11.066162 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:11.066189 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:11.067354 (+ 1165us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:11.067863 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:11.067866 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:11.067867 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:11.067912 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:11.068188 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:11.068191 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:11.068192 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:11.068348 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:11.068430 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:11.068941 (+ 511us) server_negotiation.cc:300] Negotiation successful
0201 03:02:11.069082 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260201 03:02:11.069839 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:11.059083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:34275 (local address 127.18.203.195:47773)
0201 03:02:11.059493 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:11.059529 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:11.060302 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:11.060820 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:11.060828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:11.061225 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:11.062158 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:11.062173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:11.062971 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:11.062974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:11.063436 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:11.063444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:11.063703 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:11.064278 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:11.064315 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:11.065990 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:11.067464 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:11.067469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:11.067481 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:11.067770 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:11.067989 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:11.067993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:11.067995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:11.068097 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:11.068442 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:11.068448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:11.068693 (+ 245us) client_negotiation.cc:769] Sending connection context
0201 03:02:11.068878 (+ 185us) client_negotiation.cc:240] Negotiation successful
0201 03:02:11.069157 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260201 03:02:11.070784 19767 heartbeater.cc:344] Connected to a master server at 127.18.203.254:34275
I20260201 03:02:11.071024 19767 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:11.071539 19767 heartbeater.cc:507] Master 127.18.203.254:34275 requested a full tablet report, sending...
I20260201 03:02:11.072511 19302 ts_manager.cc:194] Registered new tserver with Master: 087cd8536b7842fd800ec6052e484b49 (127.18.203.195:46595)
I20260201 03:02:11.073060 19302 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:47773
I20260201 03:02:11.077420 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 01 03:02:11 dist-test-slave-gf59 krb5kdc[19254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 01 03:02:11 dist-test-slave-gf59 krb5kdc[19254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.203.193@KRBTEST.COM
I20260201 03:02:11.101074 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:11.093082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.18.203.193:35897)
0201 03:02:11.093366 (+ 284us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:11.093373 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:11.093400 (+ 27us) server_negotiation.cc:408] Connection header received
0201 03:02:11.093464 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:11.093471 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:11.093638 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:11.093774 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:11.094589 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:11.095295 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:11.095971 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:11.096217 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:11.098096 (+ 1879us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:11.098120 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:11.098132 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:11.098160 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:11.099514 (+ 1354us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:11.099915 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:11.099920 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:11.099922 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:11.099980 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:11.100192 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:11.100196 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:11.100198 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:11.100389 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:11.100585 (+ 196us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:11.100754 (+ 169us) server_negotiation.cc:300] Negotiation successful
0201 03:02:11.100917 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
W20260201 03:02:11.101959 19409 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33478
Feb 01 03:02:11 dist-test-slave-gf59 krb5kdc[19254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 01 03:02:11 dist-test-slave-gf59 krb5kdc[19254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.193@KRBTEST.COM
I20260201 03:02:11.125715 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:11.118767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.18.203.193:35897)
0201 03:02:11.118921 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:11.118924 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:11.119006 (+ 82us) server_negotiation.cc:408] Connection header received
0201 03:02:11.119140 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:11.119143 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:11.119191 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:11.119254 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:11.120022 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:11.120518 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:11.121090 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:11.121225 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:11.123243 (+ 2018us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:11.123262 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:11.123265 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:11.123290 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:11.124598 (+ 1308us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:11.124913 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:11.124915 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:11.124916 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:11.124953 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:11.125092 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:11.125094 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:11.125095 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:11.125221 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:11.125308 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:11.125453 (+ 145us) server_negotiation.cc:300] Negotiation successful
0201 03:02:11.125548 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260201 03:02:11.126698 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 19366
I20260201 03:02:11.132977 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 19502
I20260201 03:02:11.140311 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 19638
I20260201 03:02:11.146049 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 19270
2026-02-01T03:02:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3002 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:02:11 dist-test-slave-gf59 krb5kdc[19781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:02:11 dist-test-slave-gf59 krb5kdc[19781](info): set up 2 sockets
Feb 01 03:02:11 dist-test-slave-gf59 krb5kdc[19781](info): commencing operation
krb5kdc: starting...
W20260201 03:02:12.979475 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.811s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:02:13 dist-test-slave-gf59 krb5kdc[19781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:02:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:02:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:13.113940 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:46861
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:46861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:13.217705 19797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:13.217973 19797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:13.218055 19797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:13.221282 19797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:02:13.221346 19797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:13.221380 19797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:13.221405 19797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:02:13.221437 19797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:02:13.225653 19797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:46861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:46861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:13.226681 19797 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:13.227525 19797 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:13.232931 19802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:13.232913 19805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:13.232918 19803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:13.233260 19797 server_base.cc:1060] running on GCE node
I20260201 03:02:13.233714 19797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:13.234593 19797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:13.235756 19797 hybrid_clock.cc:648] HybridClock initialized: now 1769914933235745 us; error 27 us; skew 500 ppm
Feb 01 03:02:13 dist-test-slave-gf59 krb5kdc[19781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914933, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:13.238219 19797 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:02:13.239296 19797 webserver.cc:492] Webserver started at http://127.18.203.254:36213/ using document root <none> and password file <none>
I20260201 03:02:13.239854 19797 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:13.239907 19797 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:13.240094 19797 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:13.241745 19797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "d24168e7c0204653a6cb0e5cd3dea06c"
format_stamp: "Formatted at 2026-02-01 03:02:13 on dist-test-slave-gf59"
server_key: "4da99a095eeb38461e27a3fbf4fba66d"
server_key_iv: "370c4d9928d478ead59c3387d818c1a4"
server_key_version: "encryptionkey@0"
I20260201 03:02:13.242208 19797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "d24168e7c0204653a6cb0e5cd3dea06c"
format_stamp: "Formatted at 2026-02-01 03:02:13 on dist-test-slave-gf59"
server_key: "4da99a095eeb38461e27a3fbf4fba66d"
server_key_iv: "370c4d9928d478ead59c3387d818c1a4"
server_key_version: "encryptionkey@0"
I20260201 03:02:13.245467 19797 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:02:13.247483 19812 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:13.248467 19797 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:13.248570 19797 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "d24168e7c0204653a6cb0e5cd3dea06c"
format_stamp: "Formatted at 2026-02-01 03:02:13 on dist-test-slave-gf59"
server_key: "4da99a095eeb38461e27a3fbf4fba66d"
server_key_iv: "370c4d9928d478ead59c3387d818c1a4"
server_key_version: "encryptionkey@0"
I20260201 03:02:13.248652 19797 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:13.264884 19797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:13.271051 19797 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:13.271237 19797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:13.278254 19864 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:46861 every 8 connection(s)
I20260201 03:02:13.278260 19797 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:46861
I20260201 03:02:13.279273 19797 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:02:13.281920 19865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:13.287381 19865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c: Bootstrap starting.
I20260201 03:02:13.289011 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 19797
I20260201 03:02:13.289119 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:02:13.289361 19247 external_mini_cluster.cc:1468] Setting key 6783b02374c1126c340d89d1ded18c47
I20260201 03:02:13.290021 19865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:13.290870 19865 log.cc:826] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:13.292789 19865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c: No bootstrap required, opened a new log
I20260201 03:02:13.295213 19865 raft_consensus.cc:359] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24168e7c0204653a6cb0e5cd3dea06c" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 46861 } }
I20260201 03:02:13.295393 19865 raft_consensus.cc:385] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:13.295429 19865 raft_consensus.cc:740] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24168e7c0204653a6cb0e5cd3dea06c, State: Initialized, Role: FOLLOWER
I20260201 03:02:13.295876 19865 consensus_queue.cc:260] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24168e7c0204653a6cb0e5cd3dea06c" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 46861 } }
I20260201 03:02:13.296002 19865 raft_consensus.cc:399] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:02:13.296069 19865 raft_consensus.cc:493] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:02:13.296157 19865 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [term 0 FOLLOWER]: Advancing to term 1
Feb 01 03:02:13 dist-test-slave-gf59 krb5kdc[19781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:13.296968 19865 raft_consensus.cc:515] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24168e7c0204653a6cb0e5cd3dea06c" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 46861 } }
I20260201 03:02:13.297300 19865 leader_election.cc:304] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d24168e7c0204653a6cb0e5cd3dea06c; no voters:
I20260201 03:02:13.297611 19865 leader_election.cc:290] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:02:13.297696 19870 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:13.297861 19870 raft_consensus.cc:697] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [term 1 LEADER]: Becoming Leader. State: Replica: d24168e7c0204653a6cb0e5cd3dea06c, State: Running, Role: LEADER
I20260201 03:02:13.298372 19870 consensus_queue.cc:237] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24168e7c0204653a6cb0e5cd3dea06c" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 46861 } }
I20260201 03:02:13.298767 19865 sys_catalog.cc:565] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:02:13.299552 19870 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d24168e7c0204653a6cb0e5cd3dea06c. Latest consensus state: current_term: 1 leader_uuid: "d24168e7c0204653a6cb0e5cd3dea06c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24168e7c0204653a6cb0e5cd3dea06c" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 46861 } } }
I20260201 03:02:13.299685 19870 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:13.299947 19871 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d24168e7c0204653a6cb0e5cd3dea06c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24168e7c0204653a6cb0e5cd3dea06c" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 46861 } } }
I20260201 03:02:13.300042 19871 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:13.300040 19879 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:02:13.302983 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:13.290420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.18.203.254:46861)
0201 03:02:13.290839 (+ 419us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:13.290849 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:13.290883 (+ 34us) server_negotiation.cc:408] Connection header received
0201 03:02:13.291527 (+ 644us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:13.291558 (+ 31us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:13.291943 (+ 385us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:13.292254 (+ 311us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:13.293128 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:13.293851 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:13.294488 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:13.294758 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:13.296997 (+ 2239us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:13.297015 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:13.297026 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:13.297060 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:13.299752 (+ 2692us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:13.300567 (+ 815us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:13.300574 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:13.300579 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:13.300697 (+ 118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:13.301051 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:13.301055 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:13.301057 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:13.301479 (+ 422us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:13.301709 (+ 230us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:13.301942 (+ 233us) server_negotiation.cc:300] Negotiation successful
0201 03:02:13.302211 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":132,"threads_started":1}
I20260201 03:02:13.303117 19879 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:02:13.308534 19879 catalog_manager.cc:1357] Generated new cluster ID: 146a7d77326a406985c08acbd9535494
I20260201 03:02:13.308606 19879 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:02:13.333765 19879 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:02:13.334965 19879 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:02:13.345702 19879 catalog_manager.cc:6027] T 00000000000000000000000000000000 P d24168e7c0204653a6cb0e5cd3dea06c: Generated new TSK 0
I20260201 03:02:13.346591 19879 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:13.396735 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:46861
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:13.502085 19893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:13.502326 19893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:13.502406 19893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:13.505625 19893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:13.505692 19893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:13.505795 19893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:02:13.509848 19893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:46861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:13.510859 19893 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:13.511727 19893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:13.518168 19899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:13.518170 19901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:13.518229 19898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:13.518537 19893 server_base.cc:1060] running on GCE node
I20260201 03:02:13.518865 19893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:13.519369 19893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:13.520507 19893 hybrid_clock.cc:648] HybridClock initialized: now 1769914933520479 us; error 44 us; skew 500 ppm
Feb 01 03:02:13 dist-test-slave-gf59 krb5kdc[19781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914933, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:13.522943 19893 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:02:13.524040 19893 webserver.cc:492] Webserver started at http://127.18.203.193:36723/ using document root <none> and password file <none>
I20260201 03:02:13.524564 19893 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:13.524614 19893 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:13.524794 19893 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:13.526470 19893 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "8ea37434706840c7a6cc137756278cb2"
format_stamp: "Formatted at 2026-02-01 03:02:13 on dist-test-slave-gf59"
server_key: "979427704e40a712707093e90cad5f66"
server_key_iv: "4d406dd48c61443da4e049d1862aec9d"
server_key_version: "encryptionkey@0"
I20260201 03:02:13.526937 19893 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "8ea37434706840c7a6cc137756278cb2"
format_stamp: "Formatted at 2026-02-01 03:02:13 on dist-test-slave-gf59"
server_key: "979427704e40a712707093e90cad5f66"
server_key_iv: "4d406dd48c61443da4e049d1862aec9d"
server_key_version: "encryptionkey@0"
I20260201 03:02:13.530185 19893 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260201 03:02:13.532298 19908 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:13.533257 19893 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:13.533360 19893 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "8ea37434706840c7a6cc137756278cb2"
format_stamp: "Formatted at 2026-02-01 03:02:13 on dist-test-slave-gf59"
server_key: "979427704e40a712707093e90cad5f66"
server_key_iv: "4d406dd48c61443da4e049d1862aec9d"
server_key_version: "encryptionkey@0"
I20260201 03:02:13.533440 19893 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:13.547246 19893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:13.549611 19893 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:13.549786 19893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:13.550297 19893 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:13.551132 19893 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:13.551180 19893 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:13.551230 19893 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:13.551259 19893 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:13.559634 19893 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:36529
I20260201 03:02:13.559664 20021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:36529 every 8 connection(s)
I20260201 03:02:13.560518 19893 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:02:13.562667 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 19893
I20260201 03:02:13.562762 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:02:13.563030 19247 external_mini_cluster.cc:1468] Setting key bdbe0d5a646a8d385a5ab9c32687754c
Feb 01 03:02:13 dist-test-slave-gf59 krb5kdc[19781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914933, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:13.573091 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:13.562123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:49003 (local address 127.18.203.254:46861)
0201 03:02:13.562265 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:13.562268 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:13.562924 (+ 656us) server_negotiation.cc:408] Connection header received
0201 03:02:13.563822 (+ 898us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:13.563826 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:13.563874 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:13.563958 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:13.565627 (+ 1669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:13.566319 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:13.567157 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:13.567326 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:13.569685 (+ 2359us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:13.569704 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:13.569707 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:13.569733 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:13.571073 (+ 1340us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:13.571678 (+ 605us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:13.571681 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:13.571683 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:13.571727 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:13.572030 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:13.572032 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:13.572033 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:13.572179 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:13.572246 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:13.572844 (+ 598us) server_negotiation.cc:300] Negotiation successful
0201 03:02:13.572952 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
I20260201 03:02:13.573868 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:13.562348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:46861 (local address 127.18.203.193:49003)
0201 03:02:13.562779 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:13.562821 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:13.563591 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:13.564118 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:13.564126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:13.564555 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:13.565470 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:13.565484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:13.566454 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:13.566458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:13.567046 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:13.567056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:13.567242 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:13.567806 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:13.567830 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:13.569523 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:13.571196 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:13.571202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:13.571214 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:13.571537 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:13.571808 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:13.571812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:13.571814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:13.571935 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:13.572274 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:13.572281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:13.572589 (+ 308us) client_negotiation.cc:769] Sending connection context
0201 03:02:13.572825 (+ 236us) client_negotiation.cc:240] Negotiation successful
0201 03:02:13.573110 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":104,"threads_started":1}
I20260201 03:02:13.575007 20022 heartbeater.cc:344] Connected to a master server at 127.18.203.254:46861
I20260201 03:02:13.575258 20022 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
I20260201 03:02:13.575841 20022 heartbeater.cc:507] Master 127.18.203.254:46861 requested a full tablet report, sending...
I20260201 03:02:13.577318 19829 ts_manager.cc:194] Registered new tserver with Master: 8ea37434706840c7a6cc137756278cb2 (127.18.203.193:36529)
I20260201 03:02:13.578437 19829 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:49003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:13.612756 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:46861
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:13.717584 20029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:13.717818 20029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:13.717892 20029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:13.721128 20029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:13.721194 20029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:13.721292 20029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:02:13.725441 20029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:46861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:13.726467 20029 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:13.727295 20029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:13.733497 20037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:13.733503 20034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:13.733517 20035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:13.733812 20029 server_base.cc:1060] running on GCE node
I20260201 03:02:13.734145 20029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:13.734652 20029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:13.735793 20029 hybrid_clock.cc:648] HybridClock initialized: now 1769914933735784 us; error 29 us; skew 500 ppm
Feb 01 03:02:13 dist-test-slave-gf59 krb5kdc[19781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914933, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:13.738302 20029 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:02:13.739297 20029 webserver.cc:492] Webserver started at http://127.18.203.194:34601/ using document root <none> and password file <none>
I20260201 03:02:13.739841 20029 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:13.739890 20029 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:13.740069 20029 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:13.741703 20029 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "6838f9efe259479fa8fa9bcde56d6ebf"
format_stamp: "Formatted at 2026-02-01 03:02:13 on dist-test-slave-gf59"
server_key: "122246319e0c8043064f6b56c0092b9f"
server_key_iv: "962c2eac0c87ab7117af5750c204eea3"
server_key_version: "encryptionkey@0"
I20260201 03:02:13.742169 20029 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "6838f9efe259479fa8fa9bcde56d6ebf"
format_stamp: "Formatted at 2026-02-01 03:02:13 on dist-test-slave-gf59"
server_key: "122246319e0c8043064f6b56c0092b9f"
server_key_iv: "962c2eac0c87ab7117af5750c204eea3"
server_key_version: "encryptionkey@0"
I20260201 03:02:13.745429 20029 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:02:13.747449 20044 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:13.748437 20029 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:13.748535 20029 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "6838f9efe259479fa8fa9bcde56d6ebf"
format_stamp: "Formatted at 2026-02-01 03:02:13 on dist-test-slave-gf59"
server_key: "122246319e0c8043064f6b56c0092b9f"
server_key_iv: "962c2eac0c87ab7117af5750c204eea3"
server_key_version: "encryptionkey@0"
I20260201 03:02:13.748620 20029 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:13.759295 20029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:13.761539 20029 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:13.761698 20029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:13.762187 20029 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:13.763007 20029 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:13.763062 20029 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:13.763111 20029 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:13.763137 20029 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:13.771370 20029 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:43459
I20260201 03:02:13.771404 20157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:43459 every 8 connection(s)
I20260201 03:02:13.772298 20029 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:02:13.777998 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 20029
I20260201 03:02:13.778097 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:02:13.778338 19247 external_mini_cluster.cc:1468] Setting key 38086c1bb426aa692c65417cea2301b5
Feb 01 03:02:13 dist-test-slave-gf59 krb5kdc[19781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914933, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:13.783787 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:13.773912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:38425 (local address 127.18.203.254:46861)
0201 03:02:13.774086 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:13.774091 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:13.774706 (+ 615us) server_negotiation.cc:408] Connection header received
0201 03:02:13.775479 (+ 773us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:13.775482 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:13.775531 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:13.775601 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:13.777166 (+ 1565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:13.777650 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:13.778287 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:13.778477 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:13.780460 (+ 1983us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:13.780478 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:13.780481 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:13.780509 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:13.781874 (+ 1365us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:13.782392 (+ 518us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:13.782395 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:13.782396 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:13.782433 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:13.782726 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:13.782729 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:13.782731 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:13.782920 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:13.783002 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:13.783529 (+ 527us) server_negotiation.cc:300] Negotiation successful
0201 03:02:13.783659 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260201 03:02:13.784682 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:13.774176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:46861 (local address 127.18.203.194:38425)
0201 03:02:13.774574 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:13.774610 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:13.775312 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:13.775769 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:13.775776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:13.776157 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:13.777052 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:13.777066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:13.777733 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:13.777735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:13.778181 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:13.778189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:13.778352 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:13.778903 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:13.778927 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:13.780325 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:13.781982 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:13.781987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:13.781998 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:13.782307 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:13.782502 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:13.782505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:13.782506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:13.782627 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:13.783021 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:13.783027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:13.783303 (+ 276us) client_negotiation.cc:769] Sending connection context
0201 03:02:13.783513 (+ 210us) client_negotiation.cc:240] Negotiation successful
0201 03:02:13.783830 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260201 03:02:13.785682 20158 heartbeater.cc:344] Connected to a master server at 127.18.203.254:46861
I20260201 03:02:13.785907 20158 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:13.786372 20158 heartbeater.cc:507] Master 127.18.203.254:46861 requested a full tablet report, sending...
I20260201 03:02:13.787297 19829 ts_manager.cc:194] Registered new tserver with Master: 6838f9efe259479fa8fa9bcde56d6ebf (127.18.203.194:43459)
I20260201 03:02:13.787881 19829 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:38425
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:13.822294 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:46861
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:13.926844 20165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:13.927080 20165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:13.927160 20165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:13.930414 20165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:13.930485 20165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:13.930599 20165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:02:13.934803 20165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:46861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:13.935875 20165 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:13.936686 20165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:13.943509 20171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:13.943579 20173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:13.943509 20170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:13.943853 20165 server_base.cc:1060] running on GCE node
I20260201 03:02:13.944240 20165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:13.944746 20165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:13.945902 20165 hybrid_clock.cc:648] HybridClock initialized: now 1769914933945877 us; error 38 us; skew 500 ppm
Feb 01 03:02:13 dist-test-slave-gf59 krb5kdc[19781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914933, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:13.948400 20165 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:02:13.949440 20165 webserver.cc:492] Webserver started at http://127.18.203.195:41361/ using document root <none> and password file <none>
I20260201 03:02:13.949967 20165 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:13.950019 20165 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:13.950201 20165 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:13.951951 20165 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "bc5c66530c064d648871992e7d93db58"
format_stamp: "Formatted at 2026-02-01 03:02:13 on dist-test-slave-gf59"
server_key: "1abc44dbe78d0df9edfa5d5fd7779daa"
server_key_iv: "762188394ebcf7d5fed75d6909ff07dc"
server_key_version: "encryptionkey@0"
I20260201 03:02:13.952420 20165 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "bc5c66530c064d648871992e7d93db58"
format_stamp: "Formatted at 2026-02-01 03:02:13 on dist-test-slave-gf59"
server_key: "1abc44dbe78d0df9edfa5d5fd7779daa"
server_key_iv: "762188394ebcf7d5fed75d6909ff07dc"
server_key_version: "encryptionkey@0"
I20260201 03:02:13.955787 20165 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:02:13.957930 20180 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:13.958918 20165 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:13.959021 20165 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "bc5c66530c064d648871992e7d93db58"
format_stamp: "Formatted at 2026-02-01 03:02:13 on dist-test-slave-gf59"
server_key: "1abc44dbe78d0df9edfa5d5fd7779daa"
server_key_iv: "762188394ebcf7d5fed75d6909ff07dc"
server_key_version: "encryptionkey@0"
I20260201 03:02:13.959108 20165 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:13.993821 20165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:13.996390 20165 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:13.996567 20165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:13.997123 20165 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:13.997987 20165 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:13.998034 20165 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:13.998086 20165 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:13.998111 20165 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:14.006575 20165 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:40525
I20260201 03:02:14.006608 20293 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:40525 every 8 connection(s)
I20260201 03:02:14.007485 20165 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
Feb 01 03:02:14 dist-test-slave-gf59 krb5kdc[19781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914933, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:14.017515 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 20165
I20260201 03:02:14.017642 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:02:14.017916 19247 external_mini_cluster.cc:1468] Setting key 30966ef1cda727d3c7d07775fd5db780
I20260201 03:02:14.019209 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.009176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:44911 (local address 127.18.203.254:46861)
0201 03:02:14.009343 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:14.009348 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:14.009989 (+ 641us) server_negotiation.cc:408] Connection header received
0201 03:02:14.010762 (+ 773us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:14.010765 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:14.010816 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:14.010889 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:14.012532 (+ 1643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.013052 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.013612 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.013734 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.015909 (+ 2175us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:14.015928 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:14.015930 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:14.015965 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:14.017295 (+ 1330us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:14.017797 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:14.017800 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:14.017801 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:14.017846 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:14.018141 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:14.018144 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:14.018145 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:14.018295 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:14.018381 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:14.018924 (+ 543us) server_negotiation.cc:300] Negotiation successful
0201 03:02:14.019051 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260201 03:02:14.019852 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.009430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:46861 (local address 127.18.203.195:44911)
0201 03:02:14.009850 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:14.009886 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:14.010586 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:14.011036 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:14.011044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:14.011422 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:14.012395 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:14.012409 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.013141 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.013144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:14.013535 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:14.013540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.013701 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.014211 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:14.014233 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:14.015762 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:14.017393 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:14.017398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:14.017410 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:14.017710 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:14.017928 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:14.017931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:14.017933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:14.018059 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:14.018402 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:14.018410 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:14.018674 (+ 264us) client_negotiation.cc:769] Sending connection context
0201 03:02:14.018882 (+ 208us) client_negotiation.cc:240] Negotiation successful
0201 03:02:14.019137 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20260201 03:02:14.020820 20294 heartbeater.cc:344] Connected to a master server at 127.18.203.254:46861
I20260201 03:02:14.021057 20294 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:14.021548 20294 heartbeater.cc:507] Master 127.18.203.254:46861 requested a full tablet report, sending...
I20260201 03:02:14.022449 19829 ts_manager.cc:194] Registered new tserver with Master: bc5c66530c064d648871992e7d93db58 (127.18.203.195:40525)
I20260201 03:02:14.023015 19829 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:44911
I20260201 03:02:14.031457 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260201 03:02:14.040673 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.034316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.18.203.254:46861)
0201 03:02:14.034458 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:14.034463 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:14.034569 (+ 106us) server_negotiation.cc:408] Connection header received
0201 03:02:14.034668 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:14.034670 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:14.034714 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:14.034778 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:14.035584 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.036142 (+ 558us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.036758 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.036902 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.037940 (+ 1038us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:14.037955 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:14.037958 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:14.037984 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:14.039331 (+ 1347us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:14.039645 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:14.039647 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:14.039649 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:14.039690 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:14.039969 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:14.039972 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:14.039973 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:14.040154 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:14.040289 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:14.040425 (+ 136us) server_negotiation.cc:300] Negotiation successful
0201 03:02:14.040518 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260201 03:02:14.044781 19829 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:02:14.047240 19829 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:02:14.060283 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.055965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:40525 (local address 127.0.0.1:59110)
0201 03:02:14.056311 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:14.056356 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:14.056504 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:14.057058 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:14.057064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:14.057090 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:14.057397 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:14.057406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.058747 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.058751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:14.059760 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:14.059770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.059870 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.059931 (+ 61us) client_negotiation.cc:769] Sending connection context
0201 03:02:14.060022 (+ 91us) client_negotiation.cc:240] Negotiation successful
0201 03:02:14.060119 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
I20260201 03:02:14.060714 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.056550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:43459 (local address 127.0.0.1:58612)
0201 03:02:14.056812 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:14.056825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:14.056899 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:14.058189 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:14.058193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:14.058211 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:14.058456 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:14.058462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.059518 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.059522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:14.060407 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:14.060417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.060497 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.060514 (+ 17us) client_negotiation.cc:769] Sending connection context
0201 03:02:14.060546 (+ 32us) client_negotiation.cc:240] Negotiation successful
0201 03:02:14.060598 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":89,"threads_started":1}
I20260201 03:02:14.061106 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.056073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.18.203.195:40525)
0201 03:02:14.056491 (+ 418us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:14.056498 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:14.056522 (+ 24us) server_negotiation.cc:408] Connection header received
0201 03:02:14.056652 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:14.056659 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:14.056862 (+ 203us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:14.057060 (+ 198us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:14.057547 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.058618 (+ 1071us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.059899 (+ 1281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.060584 (+ 685us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.060721 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:14.060785 (+ 64us) server_negotiation.cc:300] Negotiation successful
0201 03:02:14.060997 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":140,"threads_started":1}
I20260201 03:02:14.061658 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.056857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:36529 (local address 127.0.0.1:46334)
0201 03:02:14.057469 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:14.057481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:14.057561 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:14.058064 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:14.058067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:14.058084 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:14.058416 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:14.058425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.060199 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.060203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:14.061322 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:14.061332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.061438 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.061452 (+ 14us) client_negotiation.cc:769] Sending connection context
0201 03:02:14.061497 (+ 45us) client_negotiation.cc:240] Negotiation successful
0201 03:02:14.061549 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":89,"threads_started":1}
I20260201 03:02:14.061717 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.057385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.18.203.194:43459)
0201 03:02:14.057766 (+ 381us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:14.057772 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:14.057793 (+ 21us) server_negotiation.cc:408] Connection header received
0201 03:02:14.057867 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:14.057872 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:14.058030 (+ 158us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:14.058578 (+ 548us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:14.058722 (+ 144us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.059388 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.060704 (+ 1316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.061312 (+ 608us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.061450 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:14.061508 (+ 58us) server_negotiation.cc:300] Negotiation successful
0201 03:02:14.061583 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":121,"threads_started":1}
I20260201 03:02:14.063194 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.057371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.18.203.193:36529)
0201 03:02:14.057725 (+ 354us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:14.057730 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:14.057748 (+ 18us) server_negotiation.cc:408] Connection header received
0201 03:02:14.057811 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:14.057815 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:14.057937 (+ 122us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:14.058063 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:14.058570 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.059656 (+ 1086us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.061539 (+ 1883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.062823 (+ 1284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.062914 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:14.062976 (+ 62us) server_negotiation.cc:300] Negotiation successful
0201 03:02:14.063063 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":100,"threads_started":1}
I20260201 03:02:14.064340 20092 tablet_service.cc:1505] Processing CreateTablet for tablet a4c17f75256e475c9e3c0276adfa468a (DEFAULT_TABLE table=test-table [id=3472da20a1df4a54b099b102c882d7e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:14.064262 20228 tablet_service.cc:1505] Processing CreateTablet for tablet a4c17f75256e475c9e3c0276adfa468a (DEFAULT_TABLE table=test-table [id=3472da20a1df4a54b099b102c882d7e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:14.065191 20092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c17f75256e475c9e3c0276adfa468a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:14.065196 20228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c17f75256e475c9e3c0276adfa468a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:14.065840 19956 tablet_service.cc:1505] Processing CreateTablet for tablet a4c17f75256e475c9e3c0276adfa468a (DEFAULT_TABLE table=test-table [id=3472da20a1df4a54b099b102c882d7e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:14.066694 19956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c17f75256e475c9e3c0276adfa468a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:14.070405 20311 tablet_bootstrap.cc:492] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf: Bootstrap starting.
I20260201 03:02:14.071659 20312 tablet_bootstrap.cc:492] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58: Bootstrap starting.
I20260201 03:02:14.072346 20313 tablet_bootstrap.cc:492] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2: Bootstrap starting.
I20260201 03:02:14.073009 20311 tablet_bootstrap.cc:654] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:14.073720 20312 tablet_bootstrap.cc:654] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:14.073830 20311 log.cc:826] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:14.073985 20313 tablet_bootstrap.cc:654] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:14.074484 20312 log.cc:826] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:14.074662 20313 log.cc:826] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:14.075603 20311 tablet_bootstrap.cc:492] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf: No bootstrap required, opened a new log
I20260201 03:02:14.075814 20311 ts_tablet_manager.cc:1403] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260201 03:02:14.076186 20313 tablet_bootstrap.cc:492] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2: No bootstrap required, opened a new log
I20260201 03:02:14.076186 20312 tablet_bootstrap.cc:492] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58: No bootstrap required, opened a new log
I20260201 03:02:14.076359 20313 ts_tablet_manager.cc:1403] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260201 03:02:14.076359 20312 ts_tablet_manager.cc:1403] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260201 03:02:14.078778 20311 raft_consensus.cc:359] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5c66530c064d648871992e7d93db58" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 40525 } } peers { permanent_uuid: "6838f9efe259479fa8fa9bcde56d6ebf" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 43459 } } peers { permanent_uuid: "8ea37434706840c7a6cc137756278cb2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 36529 } }
I20260201 03:02:14.078955 20311 raft_consensus.cc:385] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:14.079015 20311 raft_consensus.cc:740] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6838f9efe259479fa8fa9bcde56d6ebf, State: Initialized, Role: FOLLOWER
I20260201 03:02:14.079244 20313 raft_consensus.cc:359] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5c66530c064d648871992e7d93db58" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 40525 } } peers { permanent_uuid: "6838f9efe259479fa8fa9bcde56d6ebf" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 43459 } } peers { permanent_uuid: "8ea37434706840c7a6cc137756278cb2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 36529 } }
I20260201 03:02:14.079442 20313 raft_consensus.cc:385] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:14.079397 20311 consensus_queue.cc:260] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5c66530c064d648871992e7d93db58" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 40525 } } peers { permanent_uuid: "6838f9efe259479fa8fa9bcde56d6ebf" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 43459 } } peers { permanent_uuid: "8ea37434706840c7a6cc137756278cb2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 36529 } }
I20260201 03:02:14.079506 20313 raft_consensus.cc:740] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea37434706840c7a6cc137756278cb2, State: Initialized, Role: FOLLOWER
I20260201 03:02:14.079675 20312 raft_consensus.cc:359] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5c66530c064d648871992e7d93db58" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 40525 } } peers { permanent_uuid: "6838f9efe259479fa8fa9bcde56d6ebf" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 43459 } } peers { permanent_uuid: "8ea37434706840c7a6cc137756278cb2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 36529 } }
I20260201 03:02:14.079867 20312 raft_consensus.cc:385] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:14.079922 20312 raft_consensus.cc:740] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc5c66530c064d648871992e7d93db58, State: Initialized, Role: FOLLOWER
I20260201 03:02:14.079972 20313 consensus_queue.cc:260] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5c66530c064d648871992e7d93db58" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 40525 } } peers { permanent_uuid: "6838f9efe259479fa8fa9bcde56d6ebf" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 43459 } } peers { permanent_uuid: "8ea37434706840c7a6cc137756278cb2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 36529 } }
I20260201 03:02:14.080155 20158 heartbeater.cc:499] Master 127.18.203.254:46861 was elected leader, sending a full tablet report...
I20260201 03:02:14.080281 20311 ts_tablet_manager.cc:1434] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260201 03:02:14.080335 20312 consensus_queue.cc:260] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5c66530c064d648871992e7d93db58" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 40525 } } peers { permanent_uuid: "6838f9efe259479fa8fa9bcde56d6ebf" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 43459 } } peers { permanent_uuid: "8ea37434706840c7a6cc137756278cb2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 36529 } }
I20260201 03:02:14.080741 20313 ts_tablet_manager.cc:1434] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260201 03:02:14.080794 20022 heartbeater.cc:499] Master 127.18.203.254:46861 was elected leader, sending a full tablet report...
I20260201 03:02:14.081077 20312 ts_tablet_manager.cc:1434] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260201 03:02:14.081404 20294 heartbeater.cc:499] Master 127.18.203.254:46861 was elected leader, sending a full tablet report...
I20260201 03:02:14.192239 20318 raft_consensus.cc:493] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:02:14.192432 20318 raft_consensus.cc:515] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5c66530c064d648871992e7d93db58" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 40525 } } peers { permanent_uuid: "6838f9efe259479fa8fa9bcde56d6ebf" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 43459 } } peers { permanent_uuid: "8ea37434706840c7a6cc137756278cb2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 36529 } }
I20260201 03:02:14.193527 20318 leader_election.cc:290] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc5c66530c064d648871992e7d93db58 (127.18.203.195:40525), 6838f9efe259479fa8fa9bcde56d6ebf (127.18.203.194:43459)
I20260201 03:02:14.196535 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.193773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:40525 (local address 127.18.203.193:54807)
0201 03:02:14.194159 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:14.194172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:14.194281 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:14.194520 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:14.194523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:14.194553 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:14.194804 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:14.194811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.195539 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.195542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:14.196144 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:14.196151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.196252 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.196274 (+ 22us) client_negotiation.cc:769] Sending connection context
0201 03:02:14.196322 (+ 48us) client_negotiation.cc:240] Negotiation successful
0201 03:02:14.196378 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":132,"threads_started":1}
I20260201 03:02:14.196719 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.193789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:43459 (local address 127.18.203.193:38235)
0201 03:02:14.194239 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:14.194250 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:14.194348 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:14.194570 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:14.194573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:14.194608 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:14.195060 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:14.195066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.195786 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.195790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:14.196354 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:14.196362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.196461 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.196479 (+ 18us) client_negotiation.cc:769] Sending connection context
0201 03:02:14.196526 (+ 47us) client_negotiation.cc:240] Negotiation successful
0201 03:02:14.196579 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"spinlock_wait_cycles":1920,"thread_start_us":128,"threads_started":1}
I20260201 03:02:14.197000 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.193898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:54807 (local address 127.18.203.195:40525)
0201 03:02:14.194034 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:14.194039 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:14.194257 (+ 218us) server_negotiation.cc:408] Connection header received
0201 03:02:14.194381 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:14.194384 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:14.194429 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:14.194494 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:14.194928 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.195426 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.196260 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.196738 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.196778 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:14.196830 (+ 52us) server_negotiation.cc:300] Negotiation successful
0201 03:02:14.196895 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260201 03:02:14.197158 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.193901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:38235 (local address 127.18.203.194:43459)
0201 03:02:14.194034 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:14.194038 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:14.194325 (+ 287us) server_negotiation.cc:408] Connection header received
0201 03:02:14.194437 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:14.194441 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:14.194484 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:14.194548 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:14.195179 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.195690 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.196463 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.196925 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.196961 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:14.197003 (+ 42us) server_negotiation.cc:300] Negotiation successful
0201 03:02:14.197047 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260201 03:02:14.197594 20248 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c17f75256e475c9e3c0276adfa468a" candidate_uuid: "8ea37434706840c7a6cc137756278cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc5c66530c064d648871992e7d93db58" is_pre_election: true
I20260201 03:02:14.197640 20112 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c17f75256e475c9e3c0276adfa468a" candidate_uuid: "8ea37434706840c7a6cc137756278cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6838f9efe259479fa8fa9bcde56d6ebf" is_pre_election: true
I20260201 03:02:14.197877 20248 raft_consensus.cc:2468] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ea37434706840c7a6cc137756278cb2 in term 0.
I20260201 03:02:14.197877 20112 raft_consensus.cc:2468] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ea37434706840c7a6cc137756278cb2 in term 0.
I20260201 03:02:14.198313 19909 leader_election.cc:304] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ea37434706840c7a6cc137756278cb2, bc5c66530c064d648871992e7d93db58; no voters:
I20260201 03:02:14.198539 20318 raft_consensus.cc:2804] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:02:14.198611 20318 raft_consensus.cc:493] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:02:14.198657 20318 raft_consensus.cc:3060] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:14.199642 20318 raft_consensus.cc:515] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5c66530c064d648871992e7d93db58" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 40525 } } peers { permanent_uuid: "6838f9efe259479fa8fa9bcde56d6ebf" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 43459 } } peers { permanent_uuid: "8ea37434706840c7a6cc137756278cb2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 36529 } }
I20260201 03:02:14.199980 20318 leader_election.cc:290] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [CANDIDATE]: Term 1 election: Requested vote from peers bc5c66530c064d648871992e7d93db58 (127.18.203.195:40525), 6838f9efe259479fa8fa9bcde56d6ebf (127.18.203.194:43459)
I20260201 03:02:14.200289 20248 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c17f75256e475c9e3c0276adfa468a" candidate_uuid: "8ea37434706840c7a6cc137756278cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc5c66530c064d648871992e7d93db58"
I20260201 03:02:14.200340 20112 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c17f75256e475c9e3c0276adfa468a" candidate_uuid: "8ea37434706840c7a6cc137756278cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6838f9efe259479fa8fa9bcde56d6ebf"
I20260201 03:02:14.200412 20248 raft_consensus.cc:3060] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:14.200450 20112 raft_consensus.cc:3060] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:14.201582 20248 raft_consensus.cc:2468] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ea37434706840c7a6cc137756278cb2 in term 1.
I20260201 03:02:14.201582 20112 raft_consensus.cc:2468] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ea37434706840c7a6cc137756278cb2 in term 1.
I20260201 03:02:14.201879 19912 leader_election.cc:304] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6838f9efe259479fa8fa9bcde56d6ebf, 8ea37434706840c7a6cc137756278cb2; no voters:
I20260201 03:02:14.202046 20318 raft_consensus.cc:2804] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:14.202253 20318 raft_consensus.cc:697] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ea37434706840c7a6cc137756278cb2, State: Running, Role: LEADER
I20260201 03:02:14.202562 20318 consensus_queue.cc:237] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5c66530c064d648871992e7d93db58" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 40525 } } peers { permanent_uuid: "6838f9efe259479fa8fa9bcde56d6ebf" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 43459 } } peers { permanent_uuid: "8ea37434706840c7a6cc137756278cb2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 36529 } }
I20260201 03:02:14.205624 19826 catalog_manager.cc:5654] T a4c17f75256e475c9e3c0276adfa468a P 8ea37434706840c7a6cc137756278cb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ea37434706840c7a6cc137756278cb2 (127.18.203.193). New cstate: current_term: 1 leader_uuid: "8ea37434706840c7a6cc137756278cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5c66530c064d648871992e7d93db58" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 40525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6838f9efe259479fa8fa9bcde56d6ebf" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 43459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ea37434706840c7a6cc137756278cb2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 36529 } health_report { overall_health: HEALTHY } } }
Feb 01 03:02:14 dist-test-slave-gf59 krb5kdc[19781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.193@KRBTEST.COM
I20260201 03:02:14.250464 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.242471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.18.203.193:36529)
0201 03:02:14.242633 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:14.242637 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:14.242747 (+ 110us) server_negotiation.cc:408] Connection header received
0201 03:02:14.242857 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:14.242860 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:14.242905 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:14.242976 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:14.243806 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.244294 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.244995 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.245140 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.247452 (+ 2312us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:14.247480 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:14.247493 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:14.247519 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:14.249114 (+ 1595us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:14.249491 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:14.249495 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:14.249497 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:14.249548 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:14.249750 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:14.249752 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:14.249753 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:14.249929 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:14.250106 (+ 177us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:14.250200 (+ 94us) server_negotiation.cc:300] Negotiation successful
0201 03:02:14.250299 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260201 03:02:14.258333 20295 tablet.cc:2378] T a4c17f75256e475c9e3c0276adfa468a P bc5c66530c064d648871992e7d93db58: Can't schedule compaction. Clean time has not been advanced past its initial value.
Feb 01 03:02:14 dist-test-slave-gf59 krb5kdc[19781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 01 03:02:14 dist-test-slave-gf59 krb5kdc[19781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.203.193@KRBTEST.COM
W20260201 03:02:14.273295 20159 tablet.cc:2378] T a4c17f75256e475c9e3c0276adfa468a P 6838f9efe259479fa8fa9bcde56d6ebf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:02:14.275157 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.267991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.18.203.193:36529)
0201 03:02:14.268129 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:14.268133 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:14.268261 (+ 128us) server_negotiation.cc:408] Connection header received
0201 03:02:14.268391 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:14.268394 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:14.268438 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:14.268506 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:14.269285 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.269742 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.270375 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.270551 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.272529 (+ 1978us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:14.272549 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:14.272551 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:14.272576 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:14.273896 (+ 1320us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:14.274251 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:14.274254 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:14.274255 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:14.274291 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:14.274442 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:14.274444 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:14.274445 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:14.274564 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:14.274665 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:14.274903 (+ 238us) server_negotiation.cc:300] Negotiation successful
0201 03:02:14.274997 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Feb 01 03:02:14 dist-test-slave-gf59 krb5kdc[19781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 01 03:02:14 dist-test-slave-gf59 krb5kdc[19781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.193@KRBTEST.COM
I20260201 03:02:14.299799 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:14.292725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.18.203.193:36529)
0201 03:02:14.292879 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:14.292884 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:14.292969 (+ 85us) server_negotiation.cc:408] Connection header received
0201 03:02:14.293125 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:14.293128 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:14.293174 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:14.293240 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:14.294059 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.294528 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:14.295167 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:14.295364 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:14.297317 (+ 1953us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:14.297335 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:14.297337 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:14.297362 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:14.298444 (+ 1082us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:14.298793 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:14.298797 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:14.298799 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:14.298842 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:14.299041 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:14.299044 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:14.299046 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:14.299172 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:14.299286 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:14.299499 (+ 213us) server_negotiation.cc:300] Negotiation successful
0201 03:02:14.299638 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260201 03:02:14.301167 19936 tablet_service.cc:3012] Rejecting scan request for tablet a4c17f75256e475c9e3c0276adfa468a: Uninitialized: clean time has not yet been initialized
I20260201 03:02:14.302754 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 19893
I20260201 03:02:14.309238 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 20029
I20260201 03:02:14.314337 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 20165
I20260201 03:02:14.319375 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 19797
2026-02-01T03:02:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3173 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:02:14 dist-test-slave-gf59 krb5kdc[20336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:02:14 dist-test-slave-gf59 krb5kdc[20336](info): set up 2 sockets
Feb 01 03:02:14 dist-test-slave-gf59 krb5kdc[20336](info): commencing operation
krb5kdc: starting...
W20260201 03:02:16.152076 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.810s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:02:16 dist-test-slave-gf59 krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:02:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:02:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:16.284493 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:34753
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:34753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:16.390726 20352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:16.390962 20352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:16.391016 20352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:16.394196 20352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:02:16.394260 20352 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260201 03:02:16.394277 20352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:16.394304 20352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:16.394327 20352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:02:16.394344 20352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:02:16.398478 20352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:34753
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:34753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:16.399487 20352 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:16.400305 20352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:16.405846 20357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:16.405917 20358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:16.405853 20360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:16.406095 20352 server_base.cc:1060] running on GCE node
I20260201 03:02:16.406682 20352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:16.407637 20352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:16.408794 20352 hybrid_clock.cc:648] HybridClock initialized: now 1769914936408789 us; error 43 us; skew 500 ppm
Feb 01 03:02:16 dist-test-slave-gf59 krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914936, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:16.411242 20352 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:02:16.412329 20352 webserver.cc:492] Webserver started at http://127.18.203.254:43015/ using document root <none> and password file <none>
I20260201 03:02:16.412870 20352 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:16.412921 20352 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:16.413103 20352 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:16.414814 20352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "841fd64639dc45919f61fa075e1f90c7"
format_stamp: "Formatted at 2026-02-01 03:02:16 on dist-test-slave-gf59"
server_key: "41ff1e339d281e1490f3bf5f7090cc26"
server_key_iv: "a2c5ec2eaed4e6746d1ab52fc823aab4"
server_key_version: "encryptionkey@0"
I20260201 03:02:16.415282 20352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "841fd64639dc45919f61fa075e1f90c7"
format_stamp: "Formatted at 2026-02-01 03:02:16 on dist-test-slave-gf59"
server_key: "41ff1e339d281e1490f3bf5f7090cc26"
server_key_iv: "a2c5ec2eaed4e6746d1ab52fc823aab4"
server_key_version: "encryptionkey@0"
I20260201 03:02:16.418557 20352 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:02:16.420755 20367 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:16.421718 20352 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:16.421831 20352 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "841fd64639dc45919f61fa075e1f90c7"
format_stamp: "Formatted at 2026-02-01 03:02:16 on dist-test-slave-gf59"
server_key: "41ff1e339d281e1490f3bf5f7090cc26"
server_key_iv: "a2c5ec2eaed4e6746d1ab52fc823aab4"
server_key_version: "encryptionkey@0"
I20260201 03:02:16.421912 20352 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:16.438263 20352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:16.440901 20352 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:16.441080 20352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:16.448941 20352 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:34753
I20260201 03:02:16.449146 20429 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:34753 every 8 connection(s)
I20260201 03:02:16.450037 20352 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:02:16.452811 20430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:16.457885 20430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7: Bootstrap starting.
I20260201 03:02:16.459467 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 20352
I20260201 03:02:16.459553 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:02:16.459784 19247 external_mini_cluster.cc:1468] Setting key 6bd53419b702343ebad995755abae60c
I20260201 03:02:16.460255 20430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:16.461071 20430 log.cc:826] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:16.463053 20430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7: No bootstrap required, opened a new log
I20260201 03:02:16.466447 20430 raft_consensus.cc:359] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841fd64639dc45919f61fa075e1f90c7" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34753 } }
I20260201 03:02:16.466693 20430 raft_consensus.cc:385] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:16.466766 20430 raft_consensus.cc:740] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841fd64639dc45919f61fa075e1f90c7, State: Initialized, Role: FOLLOWER
I20260201 03:02:16.467207 20430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841fd64639dc45919f61fa075e1f90c7" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34753 } }
I20260201 03:02:16.467322 20430 raft_consensus.cc:399] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Feb 01 03:02:16 dist-test-slave-gf59 krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:16.467379 20430 raft_consensus.cc:493] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:02:16.467460 20430 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:16.468330 20430 raft_consensus.cc:515] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841fd64639dc45919f61fa075e1f90c7" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34753 } }
I20260201 03:02:16.468714 20430 leader_election.cc:304] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 841fd64639dc45919f61fa075e1f90c7; no voters:
I20260201 03:02:16.469071 20430 leader_election.cc:290] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:02:16.469161 20435 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:16.469345 20435 raft_consensus.cc:697] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [term 1 LEADER]: Becoming Leader. State: Replica: 841fd64639dc45919f61fa075e1f90c7, State: Running, Role: LEADER
I20260201 03:02:16.469671 20435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841fd64639dc45919f61fa075e1f90c7" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34753 } }
I20260201 03:02:16.470278 20430 sys_catalog.cc:565] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:02:16.471191 20436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "841fd64639dc45919f61fa075e1f90c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841fd64639dc45919f61fa075e1f90c7" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34753 } } }
I20260201 03:02:16.471293 20436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:16.471501 20437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 841fd64639dc45919f61fa075e1f90c7. Latest consensus state: current_term: 1 leader_uuid: "841fd64639dc45919f61fa075e1f90c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841fd64639dc45919f61fa075e1f90c7" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34753 } } }
I20260201 03:02:16.471570 20437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:16.473105 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:16.460814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47962 (local address 127.18.203.254:34753)
0201 03:02:16.461259 (+ 445us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:16.461268 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:16.461303 (+ 35us) server_negotiation.cc:408] Connection header received
0201 03:02:16.461955 (+ 652us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:16.461979 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:16.462283 (+ 304us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:16.462576 (+ 293us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:16.463638 (+ 1062us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.464402 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:16.465256 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.465458 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:16.467997 (+ 2539us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:16.468017 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:16.468027 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:16.468072 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:16.470062 (+ 1990us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:16.470502 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:16.470508 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:16.470514 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:16.470609 (+ 95us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:16.470894 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:16.470897 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:16.470899 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:16.471361 (+ 462us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:16.471569 (+ 208us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:16.472092 (+ 523us) server_negotiation.cc:300] Negotiation successful
0201 03:02:16.472360 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":156,"threads_started":1}
W20260201 03:02:16.474975 20450 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260201 03:02:16.475050 20450 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260201 03:02:16.475135 20444 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:02:16.475898 20444 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:02:16.480813 20444 catalog_manager.cc:1357] Generated new cluster ID: 4780d127b29840e7bb6faf9d216eb5c7
I20260201 03:02:16.480880 20444 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:02:16.505731 20444 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:02:16.507030 20444 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:02:16.520252 20444 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7: Generated new TSK 0
I20260201 03:02:16.521220 20444 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:16.578128 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:34753
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:16.685158 20458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:16.685406 20458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:16.685474 20458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:16.688747 20458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:16.688822 20458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:16.688889 20458 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:02:16.688951 20458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:02:16.693293 20458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.203.254:34753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:16.694317 20458 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:16.695173 20458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:16.701481 20464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:16.701514 20463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:16.701481 20466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:16.701738 20458 server_base.cc:1060] running on GCE node
I20260201 03:02:16.702111 20458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:16.702639 20458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:16.703773 20458 hybrid_clock.cc:648] HybridClock initialized: now 1769914936703761 us; error 28 us; skew 500 ppm
Feb 01 03:02:16 dist-test-slave-gf59 krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914936, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:16.706221 20458 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:02:16.707267 20458 webserver.cc:492] Webserver started at http://127.18.203.193:41087/ using document root <none> and password file <none>
I20260201 03:02:16.707841 20458 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:16.707891 20458 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:16.708086 20458 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:16.709758 20458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "964aa1b2afb9490aa3857054535e2d2a"
format_stamp: "Formatted at 2026-02-01 03:02:16 on dist-test-slave-gf59"
server_key: "03bf52d272f9705df47f62162938ac97"
server_key_iv: "41f05dc852ab898221fd7554f75a5553"
server_key_version: "encryptionkey@0"
I20260201 03:02:16.710229 20458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "964aa1b2afb9490aa3857054535e2d2a"
format_stamp: "Formatted at 2026-02-01 03:02:16 on dist-test-slave-gf59"
server_key: "03bf52d272f9705df47f62162938ac97"
server_key_iv: "41f05dc852ab898221fd7554f75a5553"
server_key_version: "encryptionkey@0"
I20260201 03:02:16.713549 20458 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:02:16.715736 20473 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:16.716730 20458 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:16.716822 20458 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "964aa1b2afb9490aa3857054535e2d2a"
format_stamp: "Formatted at 2026-02-01 03:02:16 on dist-test-slave-gf59"
server_key: "03bf52d272f9705df47f62162938ac97"
server_key_iv: "41f05dc852ab898221fd7554f75a5553"
server_key_version: "encryptionkey@0"
I20260201 03:02:16.716912 20458 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:16.737313 20458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:16.740188 20458 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:16.740365 20458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:16.741853 20458 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:16.741909 20458 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:16.741948 20458 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:16.741966 20458 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:02:16 dist-test-slave-gf59 krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914936, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:16.752290 20458 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:35909
I20260201 03:02:16.752357 20588 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:35909 every 8 connection(s)
I20260201 03:02:16.753365 20458 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:02:16.753880 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:16.742547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:47911 (local address 127.18.203.254:34753)
0201 03:02:16.742713 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:16.742718 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:16.743422 (+ 704us) server_negotiation.cc:408] Connection header received
0201 03:02:16.744365 (+ 943us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:16.744369 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:16.744426 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:16.744531 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:16.746180 (+ 1649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.746978 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:16.747775 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.747977 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:16.750281 (+ 2304us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:16.750308 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:16.750312 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:16.750345 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:16.751959 (+ 1614us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:16.752480 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:16.752484 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:16.752486 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:16.752540 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:16.752824 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:16.752827 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:16.752829 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:16.753016 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:16.753118 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:16.753559 (+ 441us) server_negotiation.cc:300] Negotiation successful
0201 03:02:16.753704 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260201 03:02:16.754540 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 20458
I20260201 03:02:16.754593 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:16.742826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:34753 (local address 127.18.203.193:47911)
0201 03:02:16.743276 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:16.743317 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:16.744142 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:16.744697 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:16.744709 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:16.745246 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:16.745991 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:16.746010 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.747113 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:16.747117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:16.747648 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:16.747659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.747881 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:16.748487 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:16.748522 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:16.750095 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:16.752086 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:16.752092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:16.752103 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:16.752372 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:16.752623 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:16.752625 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:16.752627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:16.752735 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:16.753106 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:16.753111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:16.753326 (+ 215us) client_negotiation.cc:769] Sending connection context
0201 03:02:16.753509 (+ 183us) client_negotiation.cc:240] Negotiation successful
0201 03:02:16.753731 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":127,"threads_started":1}
I20260201 03:02:16.754629 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:02:16.754878 19247 external_mini_cluster.cc:1468] Setting key 299578f858d35a77de55483c031286bd
I20260201 03:02:16.755795 20589 heartbeater.cc:344] Connected to a master server at 127.18.203.254:34753
I20260201 03:02:16.755956 20589 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:16.756489 20589 heartbeater.cc:507] Master 127.18.203.254:34753 requested a full tablet report, sending...
I20260201 03:02:16.760743 20383 ts_manager.cc:194] Registered new tserver with Master: 964aa1b2afb9490aa3857054535e2d2a (127.18.203.193:35909)
I20260201 03:02:16.762058 20383 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:47911
I20260201 03:02:16.763878 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:16.757643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:34753 (local address 127.18.203.193:42737)
0201 03:02:16.757870 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:16.757883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:16.757965 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:16.758169 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:16.758171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:16.758319 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:16.758515 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:16.758520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.759215 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:16.759218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:16.759597 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:16.759642 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.759746 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:16.760257 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:16.760272 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:16.760612 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:16.762469 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:16.762472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:16.762474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:16.762693 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:16.762952 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:16.762955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:16.762957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:16.763013 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:16.763367 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:16.763370 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:16.763453 (+ 83us) client_negotiation.cc:769] Sending connection context
0201 03:02:16.763559 (+ 106us) client_negotiation.cc:240] Negotiation successful
0201 03:02:16.763702 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
I20260201 03:02:16.763912 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:16.757737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:42737 (local address 127.18.203.254:34753)
0201 03:02:16.757868 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:16.757872 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:16.757943 (+ 71us) server_negotiation.cc:408] Connection header received
0201 03:02:16.758031 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:16.758034 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:16.758077 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:16.758154 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:16.758625 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.759108 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:16.759733 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.759921 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:16.760746 (+ 825us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:16.760762 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:16.760765 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:16.760792 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:16.762360 (+ 1568us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:16.762792 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:16.762796 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:16.762798 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:16.762859 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:16.763102 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:16.763105 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:16.763107 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:16.763270 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:16.763366 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:16.763642 (+ 276us) server_negotiation.cc:300] Negotiation successful
0201 03:02:16.763756 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:16.799727 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:34753
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:16.906220 20600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:16.906440 20600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:16.906498 20600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:16.909682 20600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:16.909752 20600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:16.909806 20600 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:02:16.909844 20600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:02:16.913947 20600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.203.254:34753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:16.915004 20600 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:16.915845 20600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:16.922396 20608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:16.922396 20606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:16.922493 20605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:16.922667 20600 server_base.cc:1060] running on GCE node
I20260201 03:02:16.923106 20600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:16.923599 20600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:16.924772 20600 hybrid_clock.cc:648] HybridClock initialized: now 1769914936924760 us; error 26 us; skew 500 ppm
Feb 01 03:02:16 dist-test-slave-gf59 krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914936, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:16.927356 20600 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:02:16.928460 20600 webserver.cc:492] Webserver started at http://127.18.203.194:36313/ using document root <none> and password file <none>
I20260201 03:02:16.929004 20600 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:16.929060 20600 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:16.929248 20600 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:16.930962 20600 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "5c5f578725cb42daa4ae1b9f2893f2dc"
format_stamp: "Formatted at 2026-02-01 03:02:16 on dist-test-slave-gf59"
server_key: "f4f4214d5e0265edb0e923d16da06646"
server_key_iv: "e577d57bbe7c77683d506215ec0cac37"
server_key_version: "encryptionkey@0"
I20260201 03:02:16.931440 20600 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "5c5f578725cb42daa4ae1b9f2893f2dc"
format_stamp: "Formatted at 2026-02-01 03:02:16 on dist-test-slave-gf59"
server_key: "f4f4214d5e0265edb0e923d16da06646"
server_key_iv: "e577d57bbe7c77683d506215ec0cac37"
server_key_version: "encryptionkey@0"
I20260201 03:02:16.934795 20600 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260201 03:02:16.937024 20615 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:16.938007 20600 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:16.938113 20600 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "5c5f578725cb42daa4ae1b9f2893f2dc"
format_stamp: "Formatted at 2026-02-01 03:02:16 on dist-test-slave-gf59"
server_key: "f4f4214d5e0265edb0e923d16da06646"
server_key_iv: "e577d57bbe7c77683d506215ec0cac37"
server_key_version: "encryptionkey@0"
I20260201 03:02:16.938196 20600 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:16.957810 20600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:16.960691 20600 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:16.960860 20600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:16.962368 20600 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:16.962427 20600 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:16.962468 20600 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:16.962500 20600 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:02:16 dist-test-slave-gf59 krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914936, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:16.973367 20600 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:33733
I20260201 03:02:16.973389 20730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:33733 every 8 connection(s)
I20260201 03:02:16.974341 20600 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:02:16.974730 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:16.963129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:46457 (local address 127.18.203.254:34753)
0201 03:02:16.963283 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:16.963288 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:16.964017 (+ 729us) server_negotiation.cc:408] Connection header received
0201 03:02:16.964912 (+ 895us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:16.964916 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:16.964976 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:16.965082 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:16.966664 (+ 1582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.967292 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:16.968047 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.968237 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:16.970811 (+ 2574us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:16.970834 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:16.970837 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:16.970866 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:16.972657 (+ 1791us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:16.973256 (+ 599us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:16.973261 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:16.973263 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:16.973334 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:16.973645 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:16.973649 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:16.973651 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:16.973822 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:16.973918 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:16.974433 (+ 515us) server_negotiation.cc:300] Negotiation successful
0201 03:02:16.974562 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260201 03:02:16.975411 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:16.963389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:34753 (local address 127.18.203.194:46457)
0201 03:02:16.963856 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:16.963899 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:16.964703 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:16.965241 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:16.965252 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:16.965745 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:16.966426 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:16.966442 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.967433 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:16.967437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:16.967926 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:16.967935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.968172 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:16.968807 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:16.968845 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:16.970547 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:16.972809 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:16.972815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:16.972828 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:16.973155 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:16.973433 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:16.973437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:16.973439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:16.973558 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:16.973923 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:16.973930 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:16.974183 (+ 253us) client_negotiation.cc:769] Sending connection context
0201 03:02:16.974371 (+ 188us) client_negotiation.cc:240] Negotiation successful
0201 03:02:16.974634 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260201 03:02:16.975634 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 20600
I20260201 03:02:16.975739 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:02:16.975967 19247 external_mini_cluster.cc:1468] Setting key dede0b6774284fc79ac309fb478a4c6c
I20260201 03:02:16.977133 20731 heartbeater.cc:344] Connected to a master server at 127.18.203.254:34753
I20260201 03:02:16.977293 20731 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:16.977871 20731 heartbeater.cc:507] Master 127.18.203.254:34753 requested a full tablet report, sending...
I20260201 03:02:16.978886 20383 ts_manager.cc:194] Registered new tserver with Master: 5c5f578725cb42daa4ae1b9f2893f2dc (127.18.203.194:33733)
I20260201 03:02:16.979588 20383 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:46457
I20260201 03:02:16.985749 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:16.979367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:43239 (local address 127.18.203.254:34753)
0201 03:02:16.979487 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:16.979491 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:16.979928 (+ 437us) server_negotiation.cc:408] Connection header received
0201 03:02:16.980086 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:16.980089 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:16.980134 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:16.980206 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:16.980730 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.981210 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:16.981771 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.981942 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:16.982808 (+ 866us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:16.982824 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:16.982826 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:16.982848 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:16.984339 (+ 1491us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:16.984714 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:16.984717 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:16.984719 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:16.984762 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:16.984975 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:16.984978 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:16.984980 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:16.985162 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:16.985250 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:16.985494 (+ 244us) server_negotiation.cc:300] Negotiation successful
0201 03:02:16.985602 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260201 03:02:16.985774 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:16.979563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:34753 (local address 127.18.203.194:43239)
0201 03:02:16.979855 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:16.979870 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:16.979994 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:16.980238 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:16.980241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:16.980401 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:16.980607 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:16.980613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.981313 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:16.981316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:16.981674 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:16.981680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:16.981781 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:16.982362 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:16.982374 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:16.982659 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:16.984452 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:16.984456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:16.984459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:16.984623 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:16.984842 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:16.984845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:16.984847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:16.984892 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:16.985257 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:16.985260 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:16.985344 (+ 84us) client_negotiation.cc:769] Sending connection context
0201 03:02:16.985432 (+ 88us) client_negotiation.cc:240] Negotiation successful
0201 03:02:16.985570 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:17.024570 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:34753
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:17.130761 20742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:17.130980 20742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:17.131047 20742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:17.134187 20742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:17.134245 20742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:17.134307 20742 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:02:17.134367 20742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:02:17.138470 20742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.203.254:34753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:17.139484 20742 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:17.140318 20742 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:17.146586 20750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:17.146590 20748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:17.146739 20747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:17.147071 20742 server_base.cc:1060] running on GCE node
I20260201 03:02:17.147418 20742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:17.147960 20742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:17.149102 20742 hybrid_clock.cc:648] HybridClock initialized: now 1769914937149094 us; error 25 us; skew 500 ppm
Feb 01 03:02:17 dist-test-slave-gf59 krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914937, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:17.151481 20742 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:02:17.152506 20742 webserver.cc:492] Webserver started at http://127.18.203.195:33589/ using document root <none> and password file <none>
I20260201 03:02:17.153085 20742 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:17.153136 20742 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:17.153304 20742 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:17.154933 20742 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "7e6237757fa8498085563dfa59437e0d"
format_stamp: "Formatted at 2026-02-01 03:02:17 on dist-test-slave-gf59"
server_key: "de8300c24605f5a5a0a5fa10fe9465a4"
server_key_iv: "b009879b2134c1688028ace3c88454a2"
server_key_version: "encryptionkey@0"
I20260201 03:02:17.155391 20742 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "7e6237757fa8498085563dfa59437e0d"
format_stamp: "Formatted at 2026-02-01 03:02:17 on dist-test-slave-gf59"
server_key: "de8300c24605f5a5a0a5fa10fe9465a4"
server_key_iv: "b009879b2134c1688028ace3c88454a2"
server_key_version: "encryptionkey@0"
I20260201 03:02:17.158525 20742 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260201 03:02:17.160528 20757 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:17.161463 20742 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:17.161561 20742 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "7e6237757fa8498085563dfa59437e0d"
format_stamp: "Formatted at 2026-02-01 03:02:17 on dist-test-slave-gf59"
server_key: "de8300c24605f5a5a0a5fa10fe9465a4"
server_key_iv: "b009879b2134c1688028ace3c88454a2"
server_key_version: "encryptionkey@0"
I20260201 03:02:17.161638 20742 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:17.183370 20742 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:17.185843 20742 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:17.186013 20742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:17.187417 20742 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:17.187465 20742 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:17.187515 20742 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:17.187542 20742 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:02:17 dist-test-slave-gf59 krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914937, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:17.198109 20742 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:33591
I20260201 03:02:17.198136 20872 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:33591 every 8 connection(s)
I20260201 03:02:17.199110 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.188052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:57541 (local address 127.18.203.254:34753)
0201 03:02:17.188173 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.188178 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.188961 (+ 783us) server_negotiation.cc:408] Connection header received
0201 03:02:17.189767 (+ 806us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.189771 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.189844 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.189916 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:17.191304 (+ 1388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.191902 (+ 598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.192758 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.192959 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.195643 (+ 2684us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:17.195664 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:17.195666 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:17.195692 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:17.197257 (+ 1565us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:17.197688 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:17.197691 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:17.197693 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:17.197751 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:17.198093 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:17.198096 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:17.198099 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:17.198289 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:17.198373 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.198890 (+ 517us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.198985 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260201 03:02:17.199182 20742 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:02:17.199924 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.188393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:34753 (local address 127.18.203.195:57541)
0201 03:02:17.188829 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.188864 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.189577 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.190128 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.190137 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.190544 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:17.191164 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.191178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.192028 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.192032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.192636 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.192647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.192896 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.193647 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:17.193681 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:17.195477 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:17.197364 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:17.197368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:17.197377 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:17.197590 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:17.197869 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:17.197872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:17.197875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:17.197988 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:17.198389 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:17.198395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:17.198654 (+ 259us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.198872 (+ 218us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.199141 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":123,"threads_started":1}
I20260201 03:02:17.200301 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 20742
I20260201 03:02:17.200405 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:02:17.200676 19247 external_mini_cluster.cc:1468] Setting key f4a92ae86c2fdf8f8a8fd03ad4be4f8e
I20260201 03:02:17.201136 20873 heartbeater.cc:344] Connected to a master server at 127.18.203.254:34753
I20260201 03:02:17.201268 20873 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:17.201786 20873 heartbeater.cc:507] Master 127.18.203.254:34753 requested a full tablet report, sending...
I20260201 03:02:17.202783 20382 ts_manager.cc:194] Registered new tserver with Master: 7e6237757fa8498085563dfa59437e0d (127.18.203.195:33591)
I20260201 03:02:17.203454 20382 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:57541
I20260201 03:02:17.208778 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.203082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:34753 (local address 127.18.203.195:51879)
0201 03:02:17.203764 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.203778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.203873 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.204085 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.204087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.204236 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:17.204446 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.204451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.205364 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.205367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.205717 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.205722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.205799 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.206242 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:17.206254 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:17.206504 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:17.207813 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:17.207817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:17.207819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:17.207973 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:17.208157 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:17.208159 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:17.208161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:17.208197 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:17.208418 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:17.208420 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:17.208484 (+ 64us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.208554 (+ 70us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.208647 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":84,"threads_started":1}
I20260201 03:02:17.208775 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.203149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:51879 (local address 127.18.203.254:34753)
0201 03:02:17.203248 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.203251 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.203844 (+ 593us) server_negotiation.cc:408] Connection header received
0201 03:02:17.203943 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.203946 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.203991 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.204065 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:17.204553 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.205254 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.205815 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.205941 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.206590 (+ 649us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:17.206602 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:17.206604 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:17.206619 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:17.207709 (+ 1090us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:17.208049 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:17.208052 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:17.208054 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:17.208095 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:17.208244 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:17.208245 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:17.208247 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:17.208363 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:17.208425 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.208591 (+ 166us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.208670 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260201 03:02:17.214143 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 01 03:02:17 dist-test-slave-gf59 krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 01 03:02:17 dist-test-slave-gf59 krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:17.237808 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.231053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.18.203.254:34753)
0201 03:02:17.231202 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.231205 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.231218 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:02:17.231256 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.231259 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.231302 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.231372 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:17.232221 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.232711 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.233345 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.233542 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.235501 (+ 1959us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:17.235522 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:17.235525 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:17.235554 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:17.236675 (+ 1121us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:17.237010 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:17.237013 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:17.237014 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:17.237051 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:17.237200 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:17.237202 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:17.237204 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:17.237344 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:17.237438 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.237589 (+ 151us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.237679 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260201 03:02:17.240268 20382 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:02:17.242666 20382 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:02:17.254748 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.250600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:35909 (local address 127.0.0.1:45536)
0201 03:02:17.251652 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.251665 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.251759 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.252411 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.252415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.252432 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:17.252630 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.252636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.253561 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.253564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.254327 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.254336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.254429 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.254447 (+ 18us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.254510 (+ 63us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.254601 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":103,"threads_started":1}
I20260201 03:02:17.254881 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.250391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:33591 (local address 127.0.0.1:47406)
0201 03:02:17.251147 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.251190 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.251342 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.252102 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.252107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.252129 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:17.252435 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.252442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.253424 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.253428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.254255 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.254265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.254371 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.254428 (+ 57us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.254626 (+ 198us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.254687 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":79,"threads_started":1}
I20260201 03:02:17.255968 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.250540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.18.203.195:33591)
0201 03:02:17.250889 (+ 349us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.250896 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.251472 (+ 576us) server_negotiation.cc:408] Connection header received
0201 03:02:17.251566 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.251572 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.251796 (+ 224us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.251956 (+ 160us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:17.252562 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.253315 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.254825 (+ 1510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.255344 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.255463 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.255525 (+ 62us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.255603 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":156,"threads_started":1}
I20260201 03:02:17.256184 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.251039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.18.203.193:35909)
0201 03:02:17.251355 (+ 316us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.251362 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.251851 (+ 489us) server_negotiation.cc:408] Connection header received
0201 03:02:17.251943 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.251949 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.252124 (+ 175us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.252265 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:17.252735 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.253448 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.255199 (+ 1751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.255557 (+ 358us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.255717 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.255963 (+ 246us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.256044 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
I20260201 03:02:17.257066 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.251631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.18.203.194:33733)
0201 03:02:17.252016 (+ 385us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.252022 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.252216 (+ 194us) server_negotiation.cc:408] Connection header received
0201 03:02:17.252323 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.252327 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.252436 (+ 109us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.252555 (+ 119us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:17.253833 (+ 1278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.254833 (+ 1000us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.256235 (+ 1402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.256704 (+ 469us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.256805 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.256856 (+ 51us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.256916 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":110,"threads_started":1}
I20260201 03:02:17.257220 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.251327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:33733 (local address 127.0.0.1:46140)
0201 03:02:17.252110 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.252130 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.252219 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.253431 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.253433 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.253451 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:17.253708 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.253715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.254953 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.254957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.255816 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.255825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.255919 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.255935 (+ 16us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.255980 (+ 45us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.256034 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":93,"threads_started":1}
I20260201 03:02:17.258919 20523 tablet_service.cc:1505] Processing CreateTablet for tablet 64eb3803be254b28858e231c2dabe716 (DEFAULT_TABLE table=test-table [id=dbe384b0333745fc91328a82824252ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:17.258919 20807 tablet_service.cc:1505] Processing CreateTablet for tablet 64eb3803be254b28858e231c2dabe716 (DEFAULT_TABLE table=test-table [id=dbe384b0333745fc91328a82824252ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:17.259536 20650 tablet_service.cc:1505] Processing CreateTablet for tablet 64eb3803be254b28858e231c2dabe716 (DEFAULT_TABLE table=test-table [id=dbe384b0333745fc91328a82824252ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:17.259940 20523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64eb3803be254b28858e231c2dabe716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:17.259940 20807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64eb3803be254b28858e231c2dabe716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:17.260334 20650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64eb3803be254b28858e231c2dabe716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:17.265443 20896 tablet_bootstrap.cc:492] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc: Bootstrap starting.
I20260201 03:02:17.266633 20897 tablet_bootstrap.cc:492] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a: Bootstrap starting.
I20260201 03:02:17.267931 20898 tablet_bootstrap.cc:492] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d: Bootstrap starting.
I20260201 03:02:17.268085 20896 tablet_bootstrap.cc:654] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:17.268419 20897 tablet_bootstrap.cc:654] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:17.269121 20896 log.cc:826] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:17.269184 20897 log.cc:826] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:17.269825 20898 tablet_bootstrap.cc:654] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:17.270954 20898 log.cc:826] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:17.271081 20896 tablet_bootstrap.cc:492] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc: No bootstrap required, opened a new log
I20260201 03:02:17.271272 20896 ts_tablet_manager.cc:1403] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260201 03:02:17.272032 20897 tablet_bootstrap.cc:492] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a: No bootstrap required, opened a new log
I20260201 03:02:17.272228 20897 ts_tablet_manager.cc:1403] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260201 03:02:17.272957 20898 tablet_bootstrap.cc:492] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d: No bootstrap required, opened a new log
I20260201 03:02:17.273147 20898 ts_tablet_manager.cc:1403] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260201 03:02:17.274004 20896 raft_consensus.cc:359] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } }
I20260201 03:02:17.274200 20896 raft_consensus.cc:385] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:17.274255 20896 raft_consensus.cc:740] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c5f578725cb42daa4ae1b9f2893f2dc, State: Initialized, Role: FOLLOWER
I20260201 03:02:17.274652 20896 consensus_queue.cc:260] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } }
I20260201 03:02:17.275310 20731 heartbeater.cc:499] Master 127.18.203.254:34753 was elected leader, sending a full tablet report...
I20260201 03:02:17.275321 20897 raft_consensus.cc:359] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } }
I20260201 03:02:17.275529 20897 raft_consensus.cc:385] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:17.275589 20897 raft_consensus.cc:740] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964aa1b2afb9490aa3857054535e2d2a, State: Initialized, Role: FOLLOWER
I20260201 03:02:17.276048 20897 consensus_queue.cc:260] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } }
I20260201 03:02:17.276054 20898 raft_consensus.cc:359] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } }
I20260201 03:02:17.276226 20898 raft_consensus.cc:385] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:17.276288 20898 raft_consensus.cc:740] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6237757fa8498085563dfa59437e0d, State: Initialized, Role: FOLLOWER
I20260201 03:02:17.276670 20898 consensus_queue.cc:260] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } }
I20260201 03:02:17.276909 20897 ts_tablet_manager.cc:1434] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260201 03:02:17.277266 20589 heartbeater.cc:499] Master 127.18.203.254:34753 was elected leader, sending a full tablet report...
I20260201 03:02:17.277318 20873 heartbeater.cc:499] Master 127.18.203.254:34753 was elected leader, sending a full tablet report...
I20260201 03:02:17.277494 20898 ts_tablet_manager.cc:1434] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260201 03:02:17.279011 20896 ts_tablet_manager.cc:1434] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260201 03:02:17.405398 20903 raft_consensus.cc:493] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:02:17.405598 20903 raft_consensus.cc:515] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } }
I20260201 03:02:17.406762 20903 leader_election.cc:290] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e6237757fa8498085563dfa59437e0d (127.18.203.195:33591), 5c5f578725cb42daa4ae1b9f2893f2dc (127.18.203.194:33733)
I20260201 03:02:17.409888 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.406982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:33591 (local address 127.18.203.193:36637)
0201 03:02:17.407380 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.407392 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.407507 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.407780 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.407783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.407807 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:17.408049 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.408056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.408804 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.408807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.409418 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.409425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.409530 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.409553 (+ 23us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.409602 (+ 49us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.409666 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":184,"threads_started":1}
I20260201 03:02:17.409899 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.407003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:33733 (local address 127.18.203.193:54279)
0201 03:02:17.407455 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.407467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.407569 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.407846 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.407848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.407863 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:17.408085 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.408091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.408810 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.408813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.409415 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.409422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.409529 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.409553 (+ 24us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.409603 (+ 50us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.409663 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":152,"threads_started":1}
I20260201 03:02:17.410271 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.407111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:54279 (local address 127.18.203.194:33733)
0201 03:02:17.407237 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.407241 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.407553 (+ 312us) server_negotiation.cc:408] Connection header received
0201 03:02:17.407703 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.407706 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.407751 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.407814 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:17.408200 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.408684 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.409536 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.410005 (+ 469us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.410048 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.410109 (+ 61us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.410162 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260201 03:02:17.410265 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.407087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:36637 (local address 127.18.203.195:33591)
0201 03:02:17.407228 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.407232 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.407480 (+ 248us) server_negotiation.cc:408] Connection header received
0201 03:02:17.407632 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.407636 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.407688 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.407754 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:17.408170 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.408670 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.409539 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.410005 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.410054 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.410107 (+ 53us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.410167 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260201 03:02:17.410835 20827 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64eb3803be254b28858e231c2dabe716" candidate_uuid: "964aa1b2afb9490aa3857054535e2d2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e6237757fa8498085563dfa59437e0d" is_pre_election: true
I20260201 03:02:17.410840 20685 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64eb3803be254b28858e231c2dabe716" candidate_uuid: "964aa1b2afb9490aa3857054535e2d2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" is_pre_election: true
I20260201 03:02:17.411084 20827 raft_consensus.cc:2468] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 964aa1b2afb9490aa3857054535e2d2a in term 0.
I20260201 03:02:17.411094 20685 raft_consensus.cc:2468] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 964aa1b2afb9490aa3857054535e2d2a in term 0.
I20260201 03:02:17.411509 20477 leader_election.cc:304] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c5f578725cb42daa4ae1b9f2893f2dc, 964aa1b2afb9490aa3857054535e2d2a; no voters:
I20260201 03:02:17.411778 20903 raft_consensus.cc:2804] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:02:17.411849 20903 raft_consensus.cc:493] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:02:17.411878 20903 raft_consensus.cc:3060] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:17.412812 20903 raft_consensus.cc:515] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } }
I20260201 03:02:17.413179 20903 leader_election.cc:290] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [CANDIDATE]: Term 1 election: Requested vote from peers 7e6237757fa8498085563dfa59437e0d (127.18.203.195:33591), 5c5f578725cb42daa4ae1b9f2893f2dc (127.18.203.194:33733)
I20260201 03:02:17.413496 20827 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64eb3803be254b28858e231c2dabe716" candidate_uuid: "964aa1b2afb9490aa3857054535e2d2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e6237757fa8498085563dfa59437e0d"
I20260201 03:02:17.413568 20685 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64eb3803be254b28858e231c2dabe716" candidate_uuid: "964aa1b2afb9490aa3857054535e2d2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc"
I20260201 03:02:17.413625 20827 raft_consensus.cc:3060] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:17.413681 20685 raft_consensus.cc:3060] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:17.414850 20827 raft_consensus.cc:2468] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 964aa1b2afb9490aa3857054535e2d2a in term 1.
I20260201 03:02:17.414855 20685 raft_consensus.cc:2468] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 964aa1b2afb9490aa3857054535e2d2a in term 1.
I20260201 03:02:17.415153 20477 leader_election.cc:304] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c5f578725cb42daa4ae1b9f2893f2dc, 964aa1b2afb9490aa3857054535e2d2a; no voters:
I20260201 03:02:17.415307 20903 raft_consensus.cc:2804] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:17.415532 20903 raft_consensus.cc:697] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 1 LEADER]: Becoming Leader. State: Replica: 964aa1b2afb9490aa3857054535e2d2a, State: Running, Role: LEADER
I20260201 03:02:17.415899 20903 consensus_queue.cc:237] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } }
I20260201 03:02:17.418805 20384 catalog_manager.cc:5654] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 964aa1b2afb9490aa3857054535e2d2a (127.18.203.193). New cstate: current_term: 1 leader_uuid: "964aa1b2afb9490aa3857054535e2d2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } health_report { overall_health: UNKNOWN } } }
I20260201 03:02:17.446556 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.443887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.18.203.193:35909)
0201 03:02:17.444020 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.444024 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.444037 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:02:17.444135 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.444138 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.444193 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.444275 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:17.444572 (+ 297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.445082 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.445649 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.445776 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.445833 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:17.446275 (+ 442us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:17.446362 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.446404 (+ 42us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.446449 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260201 03:02:17.450054 20874 tablet.cc:2378] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:02:17.451977 20685 raft_consensus.cc:1275] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 1 FOLLOWER]: Refusing update from remote peer 964aa1b2afb9490aa3857054535e2d2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:17.451975 20827 raft_consensus.cc:1275] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d [term 1 FOLLOWER]: Refusing update from remote peer 964aa1b2afb9490aa3857054535e2d2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:17.452605 20903 consensus_queue.cc:1048] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:17.452989 20903 consensus_queue.cc:1048] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:17.461503 20911 mvcc.cc:204] Tried to move back new op lower bound from 7249571583796154368 to 7249571583658070016. Current Snapshot: MvccSnapshot[applied={T|T < 7249571583796154368}]
I20260201 03:02:17.463487 20912 mvcc.cc:204] Tried to move back new op lower bound from 7249571583796154368 to 7249571583658070016. Current Snapshot: MvccSnapshot[applied={T|T < 7249571583796154368}]
I20260201 03:02:17.471603 20384 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:17.471800 20384 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:17.474098 20384 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7: Sending DeleteTablet for 3 replicas of tablet 64eb3803be254b28858e231c2dabe716
I20260201 03:02:17.474910 20523 tablet_service.cc:1552] Processing DeleteTablet for tablet 64eb3803be254b28858e231c2dabe716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:17 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:45536
I20260201 03:02:17.475036 20650 tablet_service.cc:1552] Processing DeleteTablet for tablet 64eb3803be254b28858e231c2dabe716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:17 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:46140
I20260201 03:02:17.475265 20807 tablet_service.cc:1552] Processing DeleteTablet for tablet 64eb3803be254b28858e231c2dabe716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:17 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:47406
I20260201 03:02:17.475533 20921 tablet_replica.cc:333] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc: stopping tablet replica
I20260201 03:02:17.475849 20920 tablet_replica.cc:333] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a: stopping tablet replica
I20260201 03:02:17.475939 20384 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260201 03:02:17.476109 20920 raft_consensus.cc:2243] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 1 LEADER]: Raft consensus shutting down.
W20260201 03:02:17.476348 20384 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:02:17.476379 20922 tablet_replica.cc:333] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d: stopping tablet replica
I20260201 03:02:17.476456 20920 raft_consensus.cc:2272] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:17.476747 20922 raft_consensus.cc:2243] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:17.477123 20922 raft_consensus.cc:2272] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:17.477209 20921 raft_consensus.cc:2243] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:17.477510 20921 raft_consensus.cc:2272] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:17.478801 20920 ts_tablet_manager.cc:1916] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:17.478801 20921 ts_tablet_manager.cc:1916] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:17.481262 20921 ts_tablet_manager.cc:1929] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:02:17.481273 20920 ts_tablet_manager.cc:1929] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:02:17.481351 20920 log.cc:1199] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/wal/wals/64eb3803be254b28858e231c2dabe716
I20260201 03:02:17.481351 20921 log.cc:1199] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/64eb3803be254b28858e231c2dabe716
I20260201 03:02:17.481628 20921 ts_tablet_manager.cc:1950] T 64eb3803be254b28858e231c2dabe716 P 5c5f578725cb42daa4ae1b9f2893f2dc: Deleting consensus metadata
I20260201 03:02:17.481642 20920 ts_tablet_manager.cc:1950] T 64eb3803be254b28858e231c2dabe716 P 964aa1b2afb9490aa3857054535e2d2a: Deleting consensus metadata
I20260201 03:02:17.483073 20371 catalog_manager.cc:4985] TS 5c5f578725cb42daa4ae1b9f2893f2dc (127.18.203.194:33733): tablet 64eb3803be254b28858e231c2dabe716 (table test-table [id=dbe384b0333745fc91328a82824252ea]) successfully deleted
I20260201 03:02:17.483238 20371 catalog_manager.cc:4985] TS 964aa1b2afb9490aa3857054535e2d2a (127.18.203.193:35909): tablet 64eb3803be254b28858e231c2dabe716 (table test-table [id=dbe384b0333745fc91328a82824252ea]) successfully deleted
I20260201 03:02:17.483323 20807 tablet_service.cc:1505] Processing CreateTablet for tablet d9352e33186a4aabadc1239cd47561f7 (DEFAULT_TABLE table=test-table [id=08f12b06aa364d91ae13e4bd37328790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:17.483322 20650 tablet_service.cc:1505] Processing CreateTablet for tablet d9352e33186a4aabadc1239cd47561f7 (DEFAULT_TABLE table=test-table [id=08f12b06aa364d91ae13e4bd37328790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:17.483582 20807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9352e33186a4aabadc1239cd47561f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:17.483582 20650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9352e33186a4aabadc1239cd47561f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:17.484683 20922 ts_tablet_manager.cc:1916] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:17.485591 20898 tablet_bootstrap.cc:492] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d: Bootstrap starting.
I20260201 03:02:17.485591 20896 tablet_bootstrap.cc:492] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc: Bootstrap starting.
I20260201 03:02:17.486568 20898 tablet_bootstrap.cc:654] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:17.486568 20896 tablet_bootstrap.cc:654] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:17.487402 20523 tablet_service.cc:1505] Processing CreateTablet for tablet d9352e33186a4aabadc1239cd47561f7 (DEFAULT_TABLE table=test-table [id=08f12b06aa364d91ae13e4bd37328790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:17.487674 20523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9352e33186a4aabadc1239cd47561f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:17.487771 20896 tablet_bootstrap.cc:492] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc: No bootstrap required, opened a new log
I20260201 03:02:17.487774 20898 tablet_bootstrap.cc:492] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d: No bootstrap required, opened a new log
I20260201 03:02:17.487848 20898 ts_tablet_manager.cc:1403] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:17.487851 20896 ts_tablet_manager.cc:1403] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:17.488283 20898 raft_consensus.cc:359] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.488283 20896 raft_consensus.cc:359] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.488381 20896 raft_consensus.cc:385] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:17.488381 20898 raft_consensus.cc:385] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:17.488412 20898 raft_consensus.cc:740] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6237757fa8498085563dfa59437e0d, State: Initialized, Role: FOLLOWER
I20260201 03:02:17.488412 20896 raft_consensus.cc:740] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c5f578725cb42daa4ae1b9f2893f2dc, State: Initialized, Role: FOLLOWER
I20260201 03:02:17.488526 20896 consensus_queue.cc:260] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.488526 20898 consensus_queue.cc:260] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.488762 20896 ts_tablet_manager.cc:1434] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:02:17.488762 20898 ts_tablet_manager.cc:1434] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:02:17.489701 20897 tablet_bootstrap.cc:492] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a: Bootstrap starting.
I20260201 03:02:17.490659 20897 tablet_bootstrap.cc:654] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:17.491832 20922 ts_tablet_manager.cc:1929] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:02:17.491968 20922 log.cc:1199] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/64eb3803be254b28858e231c2dabe716
I20260201 03:02:17.492332 20922 ts_tablet_manager.cc:1950] T 64eb3803be254b28858e231c2dabe716 P 7e6237757fa8498085563dfa59437e0d: Deleting consensus metadata
I20260201 03:02:17.492626 20897 tablet_bootstrap.cc:492] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a: No bootstrap required, opened a new log
I20260201 03:02:17.492717 20897 ts_tablet_manager.cc:1403] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260201 03:02:17.493108 20897 raft_consensus.cc:359] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.493197 20897 raft_consensus.cc:385] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:17.493227 20897 raft_consensus.cc:740] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964aa1b2afb9490aa3857054535e2d2a, State: Initialized, Role: FOLLOWER
I20260201 03:02:17.493342 20897 consensus_queue.cc:260] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.493582 20897 ts_tablet_manager.cc:1434] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:02:17.493566 20369 catalog_manager.cc:4985] TS 7e6237757fa8498085563dfa59437e0d (127.18.203.195:33591): tablet 64eb3803be254b28858e231c2dabe716 (table test-table [id=dbe384b0333745fc91328a82824252ea]) successfully deleted
W20260201 03:02:17.504781 20590 tablet.cc:2378] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:02:17.536306 20904 raft_consensus.cc:493] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:02:17.536448 20904 raft_consensus.cc:515] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.537494 20904 leader_election.cc:290] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 964aa1b2afb9490aa3857054535e2d2a (127.18.203.193:35909), 5c5f578725cb42daa4ae1b9f2893f2dc (127.18.203.194:33733)
I20260201 03:02:17.540983 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.537716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:35909 (local address 127.18.203.195:59095)
0201 03:02:17.538006 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.538022 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.538132 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.538414 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.538417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.538433 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:17.538676 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.538685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.539694 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.539697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.540577 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.540586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.540699 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.540723 (+ 24us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.540773 (+ 50us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.540836 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260201 03:02:17.541208 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.537964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:33733 (local address 127.18.203.195:59015)
0201 03:02:17.538264 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.538277 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.538374 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.538701 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.538703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.538719 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:17.538923 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.538929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.539629 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.539633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.540577 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.540587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.540715 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.540726 (+ 11us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.540773 (+ 47us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.540829 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":88,"threads_started":1}
I20260201 03:02:17.541473 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.538163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:59015 (local address 127.18.203.194:33733)
0201 03:02:17.538277 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.538282 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.538330 (+ 48us) server_negotiation.cc:408] Connection header received
0201 03:02:17.538492 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.538495 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.538548 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.538595 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:17.539039 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.539496 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.540710 (+ 1214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.541202 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.541241 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.541294 (+ 53us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.541352 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260201 03:02:17.541630 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.537832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:59095 (local address 127.18.203.193:35909)
0201 03:02:17.537982 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.537987 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.538215 (+ 228us) server_negotiation.cc:408] Connection header received
0201 03:02:17.538269 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.538273 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.538320 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.538386 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:17.538788 (+ 402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.539558 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.540912 (+ 1354us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.541384 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.541416 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.541460 (+ 44us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.541504 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260201 03:02:17.542153 20685 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9352e33186a4aabadc1239cd47561f7" candidate_uuid: "7e6237757fa8498085563dfa59437e0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" is_pre_election: true
I20260201 03:02:17.542248 20543 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9352e33186a4aabadc1239cd47561f7" candidate_uuid: "7e6237757fa8498085563dfa59437e0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964aa1b2afb9490aa3857054535e2d2a" is_pre_election: true
I20260201 03:02:17.542328 20685 raft_consensus.cc:2468] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e6237757fa8498085563dfa59437e0d in term 0.
I20260201 03:02:17.542445 20543 raft_consensus.cc:2468] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e6237757fa8498085563dfa59437e0d in term 0.
I20260201 03:02:17.542858 20761 leader_election.cc:304] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c5f578725cb42daa4ae1b9f2893f2dc, 7e6237757fa8498085563dfa59437e0d; no voters:
I20260201 03:02:17.543095 20904 raft_consensus.cc:2804] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:02:17.543155 20904 raft_consensus.cc:493] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:02:17.543197 20904 raft_consensus.cc:3060] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:17.544126 20904 raft_consensus.cc:515] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.544468 20904 leader_election.cc:290] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [CANDIDATE]: Term 1 election: Requested vote from peers 964aa1b2afb9490aa3857054535e2d2a (127.18.203.193:35909), 5c5f578725cb42daa4ae1b9f2893f2dc (127.18.203.194:33733)
I20260201 03:02:17.544853 20543 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9352e33186a4aabadc1239cd47561f7" candidate_uuid: "7e6237757fa8498085563dfa59437e0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964aa1b2afb9490aa3857054535e2d2a"
I20260201 03:02:17.544873 20685 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9352e33186a4aabadc1239cd47561f7" candidate_uuid: "7e6237757fa8498085563dfa59437e0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc"
I20260201 03:02:17.544973 20685 raft_consensus.cc:3060] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:17.544973 20543 raft_consensus.cc:3060] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:17.546058 20543 raft_consensus.cc:2468] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e6237757fa8498085563dfa59437e0d in term 1.
I20260201 03:02:17.546067 20685 raft_consensus.cc:2468] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e6237757fa8498085563dfa59437e0d in term 1.
I20260201 03:02:17.546341 20761 leader_election.cc:304] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c5f578725cb42daa4ae1b9f2893f2dc, 7e6237757fa8498085563dfa59437e0d; no voters:
I20260201 03:02:17.546609 20904 raft_consensus.cc:2804] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:17.546813 20904 raft_consensus.cc:697] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 1 LEADER]: Becoming Leader. State: Replica: 7e6237757fa8498085563dfa59437e0d, State: Running, Role: LEADER
I20260201 03:02:17.547075 20904 consensus_queue.cc:237] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.549625 20384 catalog_manager.cc:5654] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e6237757fa8498085563dfa59437e0d (127.18.203.195). New cstate: current_term: 1 leader_uuid: "7e6237757fa8498085563dfa59437e0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } health_report { overall_health: HEALTHY } } }
Feb 01 03:02:17 dist-test-slave-gf59 krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914936, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:17.589526 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.581436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:34753 (local address 127.0.0.1:47974)
0201 03:02:17.581735 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.581751 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.581875 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.582145 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.582148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.582607 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:17.582816 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.582823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.583831 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.583834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.584198 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.584204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.584292 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.584838 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:17.584851 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:17.586503 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:17.588225 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:17.588232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:17.588237 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:17.588443 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:17.588659 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:17.588661 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:17.588663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:17.588704 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:17.589078 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:17.589084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:17.589181 (+ 97us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.589276 (+ 95us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.589385 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":110,"threads_started":1}
I20260201 03:02:17.589622 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.581530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.18.203.254:34753)
0201 03:02:17.581672 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.581677 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.581823 (+ 146us) server_negotiation.cc:408] Connection header received
0201 03:02:17.581957 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.581960 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.582012 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.582126 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:17.582942 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.583735 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.584299 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.584488 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.586624 (+ 2136us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:17.586645 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:17.586648 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:17.586675 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:17.588129 (+ 1454us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:17.588529 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:17.588532 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:17.588534 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:17.588581 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:17.588769 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:17.588772 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:17.588773 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:17.588947 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:17.589048 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.589338 (+ 290us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.589449 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260201 03:02:17.593531 20384 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:47974:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260201 03:02:17.594343 20384 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:02:17.599321 20523 tablet_service.cc:1505] Processing CreateTablet for tablet f798d3259f1e49cc8d662e11f4deac37 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=56665e0a6b4c4613b71669b3d06e4a36]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:02:17.599421 20650 tablet_service.cc:1505] Processing CreateTablet for tablet f798d3259f1e49cc8d662e11f4deac37 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=56665e0a6b4c4613b71669b3d06e4a36]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:02:17.599825 20650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f798d3259f1e49cc8d662e11f4deac37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:17.599892 20523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f798d3259f1e49cc8d662e11f4deac37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:17.600227 20807 tablet_service.cc:1505] Processing CreateTablet for tablet f798d3259f1e49cc8d662e11f4deac37 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=56665e0a6b4c4613b71669b3d06e4a36]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:02:17.600539 20807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f798d3259f1e49cc8d662e11f4deac37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:17.603071 20896 tablet_bootstrap.cc:492] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc: Bootstrap starting.
I20260201 03:02:17.603595 20898 tablet_bootstrap.cc:492] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d: Bootstrap starting.
I20260201 03:02:17.604399 20896 tablet_bootstrap.cc:654] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:17.604671 20898 tablet_bootstrap.cc:654] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:17.605116 20897 tablet_bootstrap.cc:492] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: Bootstrap starting.
I20260201 03:02:17.606237 20897 tablet_bootstrap.cc:654] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:17.606793 20898 tablet_bootstrap.cc:492] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d: No bootstrap required, opened a new log
I20260201 03:02:17.606881 20898 ts_tablet_manager.cc:1403] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260201 03:02:17.607438 20898 raft_consensus.cc:359] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.607573 20898 raft_consensus.cc:385] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:17.607666 20898 raft_consensus.cc:740] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6237757fa8498085563dfa59437e0d, State: Initialized, Role: FOLLOWER
I20260201 03:02:17.607698 20897 tablet_bootstrap.cc:492] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: No bootstrap required, opened a new log
I20260201 03:02:17.607780 20897 ts_tablet_manager.cc:1403] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260201 03:02:17.607832 20898 consensus_queue.cc:260] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.608150 20898 ts_tablet_manager.cc:1434] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260201 03:02:17.608309 20897 raft_consensus.cc:359] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.608417 20897 raft_consensus.cc:385] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:17.608453 20897 raft_consensus.cc:740] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964aa1b2afb9490aa3857054535e2d2a, State: Initialized, Role: FOLLOWER
I20260201 03:02:17.608410 20904 tablet_replica.cc:442] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } }
I20260201 03:02:17.608516 20904 tablet_replica.cc:445] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:17.609021 20896 tablet_bootstrap.cc:492] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc: No bootstrap required, opened a new log
I20260201 03:02:17.609133 20896 ts_tablet_manager.cc:1403] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260201 03:02:17.609611 20896 raft_consensus.cc:359] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.609717 20896 raft_consensus.cc:385] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:17.609752 20896 raft_consensus.cc:740] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c5f578725cb42daa4ae1b9f2893f2dc, State: Initialized, Role: FOLLOWER
I20260201 03:02:17.609887 20896 consensus_queue.cc:260] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.610113 20897 consensus_queue.cc:260] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.610445 20897 ts_tablet_manager.cc:1434] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260201 03:02:17.610754 20902 tablet_replica.cc:442] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } }
I20260201 03:02:17.610917 20902 tablet_replica.cc:445] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:17.610766 20896 ts_tablet_manager.cc:1434] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:17.610401 20916 tablet_replica.cc:442] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } }
I20260201 03:02:17.611197 20916 tablet_replica.cc:445] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:17.672935 20916 raft_consensus.cc:493] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:02:17.673094 20916 raft_consensus.cc:515] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.673553 20916 leader_election.cc:290] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c5f578725cb42daa4ae1b9f2893f2dc (127.18.203.194:33733), 7e6237757fa8498085563dfa59437e0d (127.18.203.195:33591)
I20260201 03:02:17.674006 20827 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f798d3259f1e49cc8d662e11f4deac37" candidate_uuid: "964aa1b2afb9490aa3857054535e2d2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e6237757fa8498085563dfa59437e0d" is_pre_election: true
I20260201 03:02:17.674006 20685 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f798d3259f1e49cc8d662e11f4deac37" candidate_uuid: "964aa1b2afb9490aa3857054535e2d2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" is_pre_election: true
I20260201 03:02:17.674175 20685 raft_consensus.cc:2468] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 964aa1b2afb9490aa3857054535e2d2a in term 0.
I20260201 03:02:17.674173 20827 raft_consensus.cc:2468] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 964aa1b2afb9490aa3857054535e2d2a in term 0.
I20260201 03:02:17.674470 20475 leader_election.cc:304] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e6237757fa8498085563dfa59437e0d, 964aa1b2afb9490aa3857054535e2d2a; no voters:
I20260201 03:02:17.674666 20916 raft_consensus.cc:2804] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:02:17.674731 20916 raft_consensus.cc:493] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:02:17.674759 20916 raft_consensus.cc:3060] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:17.675735 20916 raft_consensus.cc:515] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.676079 20916 leader_election.cc:290] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [CANDIDATE]: Term 1 election: Requested vote from peers 5c5f578725cb42daa4ae1b9f2893f2dc (127.18.203.194:33733), 7e6237757fa8498085563dfa59437e0d (127.18.203.195:33591)
I20260201 03:02:17.676425 20685 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f798d3259f1e49cc8d662e11f4deac37" candidate_uuid: "964aa1b2afb9490aa3857054535e2d2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc"
I20260201 03:02:17.676425 20827 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f798d3259f1e49cc8d662e11f4deac37" candidate_uuid: "964aa1b2afb9490aa3857054535e2d2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e6237757fa8498085563dfa59437e0d"
I20260201 03:02:17.676550 20685 raft_consensus.cc:3060] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:17.676550 20827 raft_consensus.cc:3060] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:17.677577 20685 raft_consensus.cc:2468] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 964aa1b2afb9490aa3857054535e2d2a in term 1.
I20260201 03:02:17.677577 20827 raft_consensus.cc:2468] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 964aa1b2afb9490aa3857054535e2d2a in term 1.
I20260201 03:02:17.677876 20475 leader_election.cc:304] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5c5f578725cb42daa4ae1b9f2893f2dc, 7e6237757fa8498085563dfa59437e0d, 964aa1b2afb9490aa3857054535e2d2a; no voters:
I20260201 03:02:17.678059 20916 raft_consensus.cc:2804] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:17.678165 20916 raft_consensus.cc:697] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [term 1 LEADER]: Becoming Leader. State: Replica: 964aa1b2afb9490aa3857054535e2d2a, State: Running, Role: LEADER
I20260201 03:02:17.678304 20916 consensus_queue.cc:237] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } }
I20260201 03:02:17.679018 20903 tablet_replica.cc:442] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: TxnStatusTablet state changed. Reason: New leader 964aa1b2afb9490aa3857054535e2d2a. Latest consensus state: current_term: 1 leader_uuid: "964aa1b2afb9490aa3857054535e2d2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } }
I20260201 03:02:17.679105 20903 tablet_replica.cc:445] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:02:17.679376 20939 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260201 03:02:17.679600 20939 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260201 03:02:17.680157 20382 catalog_manager.cc:5654] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 964aa1b2afb9490aa3857054535e2d2a (127.18.203.193). New cstate: current_term: 1 leader_uuid: "964aa1b2afb9490aa3857054535e2d2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } health_report { overall_health: UNKNOWN } } }
I20260201 03:02:17.708006 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.705333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:35909 (local address 127.0.0.1:45554)
0201 03:02:17.705496 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.705512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.705632 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.705881 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.705884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.705891 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:02:17.706082 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.706089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.706758 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.706762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.707182 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.707188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.707292 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.707343 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:02:17.707711 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:02:17.707729 (+ 18us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.707789 (+ 60us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.707875 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260201 03:02:17.708006 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.705430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.18.203.193:35909)
0201 03:02:17.705552 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.705556 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.705570 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:02:17.705743 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.705746 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.705795 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.705859 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:17.706187 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.706655 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.707290 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.707473 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.707514 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:17.707584 (+ 70us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:17.707693 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.707808 (+ 115us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.707860 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260201 03:02:17.710717 20685 raft_consensus.cc:1275] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 1 FOLLOWER]: Refusing update from remote peer 964aa1b2afb9490aa3857054535e2d2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:17.710717 20827 raft_consensus.cc:1275] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d [term 1 FOLLOWER]: Refusing update from remote peer 964aa1b2afb9490aa3857054535e2d2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:17.711156 20903 consensus_queue.cc:1048] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:17.711303 20916 consensus_queue.cc:1048] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:17.712661 20902 tablet_replica.cc:442] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc: TxnStatusTablet state changed. Reason: New leader 964aa1b2afb9490aa3857054535e2d2a. Latest consensus state: current_term: 1 leader_uuid: "964aa1b2afb9490aa3857054535e2d2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } }
I20260201 03:02:17.712690 20904 tablet_replica.cc:442] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d: TxnStatusTablet state changed. Reason: New leader 964aa1b2afb9490aa3857054535e2d2a. Latest consensus state: current_term: 1 leader_uuid: "964aa1b2afb9490aa3857054535e2d2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } }
I20260201 03:02:17.712777 20904 tablet_replica.cc:445] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:17.712777 20902 tablet_replica.cc:445] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:17.714560 20907 tablet_replica.cc:442] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "964aa1b2afb9490aa3857054535e2d2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } }
I20260201 03:02:17.714679 20907 tablet_replica.cc:445] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:02:17.714777 20910 mvcc.cc:204] Tried to move back new op lower bound from 7249571584858832896 to 7249571584731561984. Current Snapshot: MvccSnapshot[applied={T|T < 7249571584858832896}]
I20260201 03:02:17.715534 20904 tablet_replica.cc:442] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "964aa1b2afb9490aa3857054535e2d2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } }
I20260201 03:02:17.715677 20904 tablet_replica.cc:445] T f798d3259f1e49cc8d662e11f4deac37 P 7e6237757fa8498085563dfa59437e0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:17.714566 20903 tablet_replica.cc:442] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "964aa1b2afb9490aa3857054535e2d2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } }
I20260201 03:02:17.716001 20903 tablet_replica.cc:445] T f798d3259f1e49cc8d662e11f4deac37 P 964aa1b2afb9490aa3857054535e2d2a: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:02:17.716833 20902 tablet_replica.cc:442] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "964aa1b2afb9490aa3857054535e2d2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 } } peers { permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 } } peers { permanent_uuid: "7e6237757fa8498085563dfa59437e0d" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33591 } } }
I20260201 03:02:17.716953 20902 tablet_replica.cc:445] T f798d3259f1e49cc8d662e11f4deac37 P 5c5f578725cb42daa4ae1b9f2893f2dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:17.724455 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.721061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.18.203.195:33591)
0201 03:02:17.721167 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.721170 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.721233 (+ 63us) server_negotiation.cc:408] Connection header received
0201 03:02:17.721384 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.721387 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.721437 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.721526 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:17.721923 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.722399 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.723263 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.723391 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.723440 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:17.724096 (+ 656us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:17.724187 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.724276 (+ 89us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.724327 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260201 03:02:17.726075 20732 tablet.cc:2378] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:02:17.734213 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.731429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:39531 (local address 127.18.203.193:35909)
0201 03:02:17.731542 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.731545 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.731781 (+ 236us) server_negotiation.cc:408] Connection header received
0201 03:02:17.731903 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.731905 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.731951 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.732012 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:17.732406 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.732854 (+ 448us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.733703 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.733828 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.733862 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:17.733924 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:17.733975 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.734067 (+ 92us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.734108 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260201 03:02:17.734279 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.731342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:35909 (local address 127.18.203.195:39531)
0201 03:02:17.731708 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.731723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.731823 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.732032 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.732035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.732050 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:02:17.732322 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.732328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.732939 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.732943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.733612 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.733620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.733727 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.733768 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:02:17.733993 (+ 225us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:02:17.733998 (+ 5us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.734063 (+ 65us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.734128 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
I20260201 03:02:17.740351 20685 raft_consensus.cc:1275] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 1 FOLLOWER]: Refusing update from remote peer 7e6237757fa8498085563dfa59437e0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:17.740603 20543 raft_consensus.cc:1275] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a [term 1 FOLLOWER]: Refusing update from remote peer 7e6237757fa8498085563dfa59437e0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:17.741112 20927 consensus_queue.cc:1048] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c5f578725cb42daa4ae1b9f2893f2dc" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:17.741281 20904 consensus_queue.cc:1048] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "964aa1b2afb9490aa3857054535e2d2a" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:17.768558 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.758747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:48867 (local address 127.18.203.195:33591)
0201 03:02:17.759002 (+ 255us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.759006 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.764517 (+ 5511us) server_negotiation.cc:408] Connection header received
0201 03:02:17.764683 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.764687 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.764735 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.764776 (+ 41us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:17.765463 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.766677 (+ 1214us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.767676 (+ 999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.767868 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.768090 (+ 222us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:17.768174 (+ 84us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:17.768219 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.768366 (+ 147us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.768417 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260201 03:02:17.768970 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.758401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:33591 (local address 127.18.203.193:48867)
0201 03:02:17.764441 (+ 6040us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:17.764456 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:17.764585 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:17.765010 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:17.765013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:17.765022 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:02:17.765342 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.765350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.766846 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.766849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:17.767527 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:17.767536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.767935 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.767978 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:02:17.768275 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:02:17.768280 (+ 5us) client_negotiation.cc:769] Sending connection context
0201 03:02:17.768756 (+ 476us) client_negotiation.cc:240] Negotiation successful
0201 03:02:17.768843 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5902,"thread_start_us":99,"threads_started":1}
I20260201 03:02:17.801183 20384 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:17.801369 20384 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:17.803789 20384 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 841fd64639dc45919f61fa075e1f90c7: Sending DeleteTablet for 3 replicas of tablet d9352e33186a4aabadc1239cd47561f7
I20260201 03:02:17.804858 20807 tablet_service.cc:1552] Processing DeleteTablet for tablet d9352e33186a4aabadc1239cd47561f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:17 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:47406
I20260201 03:02:17.805297 20922 tablet_replica.cc:333] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d: stopping tablet replica
I20260201 03:02:17.805480 20922 raft_consensus.cc:2243] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:02:17.805933 20922 raft_consensus.cc:2272] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:17.805968 20650 tablet_service.cc:1552] Processing DeleteTablet for tablet d9352e33186a4aabadc1239cd47561f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:17 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:46140
I20260201 03:02:17.806154 20921 tablet_replica.cc:333] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc: stopping tablet replica
I20260201 03:02:17.806254 20921 raft_consensus.cc:2243] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:17.806380 20921 raft_consensus.cc:2272] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:17.806784 20922 ts_tablet_manager.cc:1916] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:17.806973 20523 tablet_service.cc:1552] Processing DeleteTablet for tablet d9352e33186a4aabadc1239cd47561f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:17 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:45536
I20260201 03:02:17.807156 20920 tablet_replica.cc:333] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a: stopping tablet replica
I20260201 03:02:17.807265 20920 raft_consensus.cc:2243] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:17.807367 20920 raft_consensus.cc:2272] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:17.808277 20921 ts_tablet_manager.cc:1916] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:17.810729 20922 ts_tablet_manager.cc:1929] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:02:17.810798 20922 log.cc:1199] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/d9352e33186a4aabadc1239cd47561f7
I20260201 03:02:17.810900 20921 ts_tablet_manager.cc:1929] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:02:17.810968 20921 log.cc:1199] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/d9352e33186a4aabadc1239cd47561f7
I20260201 03:02:17.811038 20922 ts_tablet_manager.cc:1950] T d9352e33186a4aabadc1239cd47561f7 P 7e6237757fa8498085563dfa59437e0d: Deleting consensus metadata
I20260201 03:02:17.811211 20921 ts_tablet_manager.cc:1950] T d9352e33186a4aabadc1239cd47561f7 P 5c5f578725cb42daa4ae1b9f2893f2dc: Deleting consensus metadata
I20260201 03:02:17.811839 20369 catalog_manager.cc:4985] TS 7e6237757fa8498085563dfa59437e0d (127.18.203.195:33591): tablet d9352e33186a4aabadc1239cd47561f7 (table test-table [id=08f12b06aa364d91ae13e4bd37328790]) successfully deleted
I20260201 03:02:17.811911 20371 catalog_manager.cc:4985] TS 5c5f578725cb42daa4ae1b9f2893f2dc (127.18.203.194:33733): tablet d9352e33186a4aabadc1239cd47561f7 (table test-table [id=08f12b06aa364d91ae13e4bd37328790]) successfully deleted
I20260201 03:02:17.812706 20920 ts_tablet_manager.cc:1916] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:17.815264 20920 ts_tablet_manager.cc:1929] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:02:17.815349 20920 log.cc:1199] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769914928154323-19247-0/minicluster-data/ts-0/wal/wals/d9352e33186a4aabadc1239cd47561f7
I20260201 03:02:17.815598 20920 ts_tablet_manager.cc:1950] T d9352e33186a4aabadc1239cd47561f7 P 964aa1b2afb9490aa3857054535e2d2a: Deleting consensus metadata
I20260201 03:02:17.816354 20371 catalog_manager.cc:4985] TS 964aa1b2afb9490aa3857054535e2d2a (127.18.203.193:35909): tablet d9352e33186a4aabadc1239cd47561f7 (table test-table [id=08f12b06aa364d91ae13e4bd37328790]) successfully deleted
Feb 01 03:02:17 dist-test-slave-gf59 krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.203.193@KRBTEST.COM
I20260201 03:02:17.824651 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.814125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45568 (local address 127.18.203.193:35909)
0201 03:02:17.814696 (+ 571us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.814701 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.814726 (+ 25us) server_negotiation.cc:408] Connection header received
0201 03:02:17.814764 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.814767 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.814813 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.814879 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:17.816202 (+ 1323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.816905 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.817858 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.818068 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.820727 (+ 2659us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:17.820758 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:17.820774 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:17.820808 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:17.822923 (+ 2115us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:17.823403 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:17.823409 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:17.823411 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:17.823482 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:17.823745 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:17.823749 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:17.823751 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:17.823985 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:17.824194 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.824355 (+ 161us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.824475 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485}
W20260201 03:02:17.825119 20583 server_base.cc:1142] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45568
I20260201 03:02:17.835470 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:17.827359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.18.203.254:34753)
0201 03:02:17.827528 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:17.827533 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:17.827729 (+ 196us) server_negotiation.cc:408] Connection header received
0201 03:02:17.827849 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:17.827853 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:17.827909 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:17.827991 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:17.828971 (+ 980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.829773 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:17.830637 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:17.831168 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:17.832142 (+ 974us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:17.832164 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:17.832168 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:17.832195 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:17.833878 (+ 1683us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:17.834335 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:17.834338 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:17.834341 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:17.834393 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:17.834619 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:17.834622 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:17.834624 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:17.834816 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:17.834941 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:17.835162 (+ 221us) server_negotiation.cc:300] Negotiation successful
0201 03:02:17.835283 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260201 03:02:17.836014 20384 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47980
I20260201 03:02:17.837638 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 20458
I20260201 03:02:17.849382 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 20600
I20260201 03:02:17.855501 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 20742
I20260201 03:02:17.861629 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 20352
2026-02-01T03:02:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3543 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:02:17 dist-test-slave-gf59 krb5kdc[20959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:02:17 dist-test-slave-gf59 krb5kdc[20959](info): set up 2 sockets
Feb 01 03:02:17 dist-test-slave-gf59 krb5kdc[20959](info): commencing operation
krb5kdc: starting...
W20260201 03:02:19.699740 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.815s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/test-user.keytab.
W20260201 03:02:20.508368 19247 mini_kdc.cc:290] Time spent creating keytab for test-user: real 0.655s user 0.002s sys 0.001s
Feb 01 03:02:20 dist-test-slave-gf59 krb5kdc[20959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:02:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:02:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:20.606215 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:35461
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:35461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:20.713344 20975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:20.713588 20975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:20.713651 20975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:20.716914 20975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:02:20.716982 20975 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260201 03:02:20.717002 20975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:20.717039 20975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:20.717070 20975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:02:20.717100 20975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:02:20.721336 20975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:35461
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:35461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:20.722391 20975 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:20.723255 20975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:20.729465 20981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:20.730441 20975 server_base.cc:1060] running on GCE node
W20260201 03:02:20.732978 20980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:20.734088 20983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:20.734747 20975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:20.735930 20975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:20.737092 20975 hybrid_clock.cc:648] HybridClock initialized: now 1769914940737069 us; error 46 us; skew 500 ppm
Feb 01 03:02:20 dist-test-slave-gf59 krb5kdc[20959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914940, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:20.739629 20975 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:02:20.740686 20975 webserver.cc:492] Webserver started at http://127.18.203.254:43065/ using document root <none> and password file <none>
I20260201 03:02:20.741230 20975 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:20.741281 20975 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:20.741456 20975 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:20.743160 20975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "3b7655de594a446a817e936d5252660b"
format_stamp: "Formatted at 2026-02-01 03:02:20 on dist-test-slave-gf59"
server_key: "d9f6570950d32fa84cae58689c38c2fa"
server_key_iv: "58b48235e97de286d15fffeb33072cd2"
server_key_version: "encryptionkey@0"
I20260201 03:02:20.743656 20975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "3b7655de594a446a817e936d5252660b"
format_stamp: "Formatted at 2026-02-01 03:02:20 on dist-test-slave-gf59"
server_key: "d9f6570950d32fa84cae58689c38c2fa"
server_key_iv: "58b48235e97de286d15fffeb33072cd2"
server_key_version: "encryptionkey@0"
I20260201 03:02:20.746873 20975 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260201 03:02:20.749011 20990 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:20.749965 20975 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:20.750077 20975 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "3b7655de594a446a817e936d5252660b"
format_stamp: "Formatted at 2026-02-01 03:02:20 on dist-test-slave-gf59"
server_key: "d9f6570950d32fa84cae58689c38c2fa"
server_key_iv: "58b48235e97de286d15fffeb33072cd2"
server_key_version: "encryptionkey@0"
I20260201 03:02:20.750188 20975 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:20.760677 20975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:20.766381 20975 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:20.766553 20975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:20.774209 20975 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:35461
I20260201 03:02:20.774415 21052 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:35461 every 8 connection(s)
I20260201 03:02:20.775179 20975 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:02:20.777808 21053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:20.782650 21053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b: Bootstrap starting.
I20260201 03:02:20.783406 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 20975
I20260201 03:02:20.783527 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:02:20.783830 19247 external_mini_cluster.cc:1468] Setting key f3dc7d237af9058266847242b612e8d0
I20260201 03:02:20.785275 21053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:20.785933 21053 log.cc:826] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:20.787945 21053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b: No bootstrap required, opened a new log
I20260201 03:02:20.790900 21053 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7655de594a446a817e936d5252660b" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35461 } }
I20260201 03:02:20.791077 21053 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:20.791121 21053 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7655de594a446a817e936d5252660b, State: Initialized, Role: FOLLOWER
Feb 01 03:02:20 dist-test-slave-gf59 krb5kdc[20959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:20.791538 21053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7655de594a446a817e936d5252660b" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35461 } }
I20260201 03:02:20.791683 21053 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:02:20.791760 21053 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:02:20.791888 21053 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:20.793076 21053 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7655de594a446a817e936d5252660b" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35461 } }
I20260201 03:02:20.793390 21053 leader_election.cc:304] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b7655de594a446a817e936d5252660b; no voters:
I20260201 03:02:20.793660 21053 leader_election.cc:290] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:02:20.793746 21058 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:20.793932 21058 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [term 1 LEADER]: Becoming Leader. State: Replica: 3b7655de594a446a817e936d5252660b, State: Running, Role: LEADER
I20260201 03:02:20.794272 21058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7655de594a446a817e936d5252660b" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35461 } }
I20260201 03:02:20.794808 21053 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:02:20.795390 21058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b7655de594a446a817e936d5252660b. Latest consensus state: current_term: 1 leader_uuid: "3b7655de594a446a817e936d5252660b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7655de594a446a817e936d5252660b" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35461 } } }
I20260201 03:02:20.795522 21058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:20.796059 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b7655de594a446a817e936d5252660b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7655de594a446a817e936d5252660b" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35461 } } }
I20260201 03:02:20.796206 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:20.796304 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:20.784894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.18.203.254:35461)
0201 03:02:20.785319 (+ 425us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:20.785331 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:20.785372 (+ 41us) server_negotiation.cc:408] Connection header received
0201 03:02:20.785929 (+ 557us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:20.785948 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:20.786259 (+ 311us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:20.786610 (+ 351us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:20.787574 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:20.788407 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:20.789236 (+ 829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:20.789448 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:20.791860 (+ 2412us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:20.791878 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:20.791888 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:20.791920 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:20.793554 (+ 1634us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:20.794016 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:20.794022 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:20.794027 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:20.794119 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:20.794335 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:20.794339 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:20.794341 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:20.794701 (+ 360us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:20.794878 (+ 177us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:20.795196 (+ 318us) server_negotiation.cc:300] Negotiation successful
0201 03:02:20.795456 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":141,"threads_started":1}
I20260201 03:02:20.796504 21063 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:02:20.799584 21063 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:02:20.805251 21063 catalog_manager.cc:1357] Generated new cluster ID: ee3106a414e84da7924b0523a192bfc0
I20260201 03:02:20.805320 21063 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:02:20.810027 21063 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:02:20.811230 21063 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:02:20.821142 21063 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b: Generated new TSK 0
I20260201 03:02:20.822003 21063 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:20.879793 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:35461
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:20.987867 21081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:20.988095 21081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:20.988163 21081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:20.991400 21081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:20.991467 21081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:20.991520 21081 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260201 03:02:20.991541 21081 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:02:20.991581 21081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:02:20.995904 21081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.203.254:35461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:20.997097 21081 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:20.997932 21081 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:21.004362 21086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:21.004403 21087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:21.004362 21089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:21.004652 21081 server_base.cc:1060] running on GCE node
I20260201 03:02:21.005007 21081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:21.005537 21081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:21.006686 21081 hybrid_clock.cc:648] HybridClock initialized: now 1769914941006676 us; error 30 us; skew 500 ppm
Feb 01 03:02:21 dist-test-slave-gf59 krb5kdc[20959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914941, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:21.009274 21081 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:02:21.010352 21081 webserver.cc:492] Webserver started at http://127.18.203.193:36161/ using document root <none> and password file <none>
I20260201 03:02:21.010901 21081 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:21.010954 21081 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:21.011138 21081 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:21.012895 21081 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "f7cdef52faff47e6bae3b60988f108e7"
format_stamp: "Formatted at 2026-02-01 03:02:21 on dist-test-slave-gf59"
server_key: "288ba36dc45d5a949805002cd57e5d87"
server_key_iv: "fc70759b72690ff461e2f5e62330201f"
server_key_version: "encryptionkey@0"
I20260201 03:02:21.013367 21081 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "f7cdef52faff47e6bae3b60988f108e7"
format_stamp: "Formatted at 2026-02-01 03:02:21 on dist-test-slave-gf59"
server_key: "288ba36dc45d5a949805002cd57e5d87"
server_key_iv: "fc70759b72690ff461e2f5e62330201f"
server_key_version: "encryptionkey@0"
I20260201 03:02:21.017275 21081 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260201 03:02:21.019821 21096 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:21.020943 21081 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:21.021046 21081 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "f7cdef52faff47e6bae3b60988f108e7"
format_stamp: "Formatted at 2026-02-01 03:02:21 on dist-test-slave-gf59"
server_key: "288ba36dc45d5a949805002cd57e5d87"
server_key_iv: "fc70759b72690ff461e2f5e62330201f"
server_key_version: "encryptionkey@0"
I20260201 03:02:21.021148 21081 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:21.035468 21081 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:21.038291 21081 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:21.038461 21081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:21.039924 21081 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:21.039976 21081 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:21.040027 21081 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:21.040051 21081 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:02:21 dist-test-slave-gf59 krb5kdc[20959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914941, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:21.050793 21081 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:33429
I20260201 03:02:21.051906 21081 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:02:21.052775 21211 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:33429 every 8 connection(s)
I20260201 03:02:21.053792 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.040707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:56753 (local address 127.18.203.254:35461)
0201 03:02:21.040863 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.040868 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.041655 (+ 787us) server_negotiation.cc:408] Connection header received
0201 03:02:21.042754 (+ 1099us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.042759 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.042827 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.042888 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:21.044875 (+ 1987us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.045689 (+ 814us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.046583 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.046800 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.049753 (+ 2953us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:21.049782 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:21.049786 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:21.049820 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:21.051573 (+ 1753us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.052220 (+ 647us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.052224 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.052226 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.052296 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.052616 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.052620 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.052622 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.052839 (+ 217us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:21.052939 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.053467 (+ 528us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.053619 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260201 03:02:21.054262 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.041050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35461 (local address 127.18.203.193:56753)
0201 03:02:21.041498 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.041540 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.042475 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.043328 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.043339 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.043912 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:21.044688 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.044705 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.045813 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.045817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.046446 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.046456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.046687 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.047533 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:21.047564 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:21.049551 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:21.051759 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.051764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.051776 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.052109 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.052416 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.052419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.052422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.052534 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.052991 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:21.052996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:21.053212 (+ 216us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.053379 (+ 167us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.053586 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":132,"threads_started":1}
I20260201 03:02:21.055486 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 21081
I20260201 03:02:21.055660 21212 heartbeater.cc:344] Connected to a master server at 127.18.203.254:35461
I20260201 03:02:21.055831 21212 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:21.056407 21212 heartbeater.cc:507] Master 127.18.203.254:35461 requested a full tablet report, sending...
I20260201 03:02:21.056916 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:02:21.057219 19247 external_mini_cluster.cc:1468] Setting key 02a18947ee7770beb22f2a06ff5477ad
I20260201 03:02:21.058226 21006 ts_manager.cc:194] Registered new tserver with Master: f7cdef52faff47e6bae3b60988f108e7 (127.18.203.193:33429)
I20260201 03:02:21.059669 21006 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:56753
I20260201 03:02:21.066874 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.057936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35461 (local address 127.18.203.193:40097)
0201 03:02:21.058198 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.058213 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.058312 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.058596 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.058599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.058797 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:21.059089 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.059096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.060174 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.060177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.060761 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.060770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.061192 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.063743 (+ 2551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:21.063756 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:21.064046 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:21.065722 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.065725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.065727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.065888 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.066115 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.066117 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.066119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.066155 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.066479 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:21.066481 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:21.066548 (+ 67us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.066619 (+ 71us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.066718 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
I20260201 03:02:21.066964 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.057720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:40097 (local address 127.18.203.254:35461)
0201 03:02:21.057849 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.057853 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.058286 (+ 433us) server_negotiation.cc:408] Connection header received
0201 03:02:21.058389 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.058393 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.058434 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.058495 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:21.059218 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.060036 (+ 818us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.060886 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.061086 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.064154 (+ 3068us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:21.064175 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:21.064178 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:21.064205 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:21.065615 (+ 1410us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.065972 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.065975 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.065977 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.066035 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.066222 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.066225 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.066227 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.066400 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:21.066501 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.066689 (+ 188us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.066809 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:21.113852 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:35461
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:21.219533 21223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:21.219810 21223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:21.219888 21223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:21.223138 21223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:21.223208 21223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:21.223284 21223 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260201 03:02:21.223309 21223 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:02:21.223359 21223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:02:21.227638 21223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.203.254:35461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:21.228765 21223 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:21.229611 21223 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:21.236047 21231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:21.236052 21229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:21.236048 21228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:21.236347 21223 server_base.cc:1060] running on GCE node
I20260201 03:02:21.236825 21223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:21.237363 21223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:21.238534 21223 hybrid_clock.cc:648] HybridClock initialized: now 1769914941238516 us; error 31 us; skew 500 ppm
Feb 01 03:02:21 dist-test-slave-gf59 krb5kdc[20959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914941, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:21.241041 21223 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:02:21.242056 21223 webserver.cc:492] Webserver started at http://127.18.203.194:39897/ using document root <none> and password file <none>
I20260201 03:02:21.242602 21223 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:21.242653 21223 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:21.242827 21223 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:21.244590 21223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "f3c41a09ad844a85982381d531c0d95e"
format_stamp: "Formatted at 2026-02-01 03:02:21 on dist-test-slave-gf59"
server_key: "82c8c563e5ed379a2c7ec6ec7b821f49"
server_key_iv: "85c46b2fde66edc1391606704ea6104b"
server_key_version: "encryptionkey@0"
I20260201 03:02:21.245045 21223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "f3c41a09ad844a85982381d531c0d95e"
format_stamp: "Formatted at 2026-02-01 03:02:21 on dist-test-slave-gf59"
server_key: "82c8c563e5ed379a2c7ec6ec7b821f49"
server_key_iv: "85c46b2fde66edc1391606704ea6104b"
server_key_version: "encryptionkey@0"
I20260201 03:02:21.248263 21223 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:02:21.250337 21238 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:21.251312 21223 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:21.251415 21223 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "f3c41a09ad844a85982381d531c0d95e"
format_stamp: "Formatted at 2026-02-01 03:02:21 on dist-test-slave-gf59"
server_key: "82c8c563e5ed379a2c7ec6ec7b821f49"
server_key_iv: "85c46b2fde66edc1391606704ea6104b"
server_key_version: "encryptionkey@0"
I20260201 03:02:21.251497 21223 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:21.271126 21223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:21.273669 21223 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:21.273847 21223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:21.275298 21223 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:21.275350 21223 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:21.275401 21223 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:21.275432 21223 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:02:21 dist-test-slave-gf59 krb5kdc[20959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914941, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:21.286319 21223 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:41659
I20260201 03:02:21.286360 21353 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:41659 every 8 connection(s)
I20260201 03:02:21.287362 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.275968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:45685 (local address 127.18.203.254:35461)
0201 03:02:21.276097 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.276102 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.276909 (+ 807us) server_negotiation.cc:408] Connection header received
0201 03:02:21.277741 (+ 832us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.277745 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.277805 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.277880 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:21.279334 (+ 1454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.279988 (+ 654us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.280630 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.280812 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.283660 (+ 2848us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:21.283686 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:21.283689 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:21.283722 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:21.285376 (+ 1654us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.285847 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.285851 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.285853 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.285910 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.286268 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.286272 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.286274 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.286486 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:21.286574 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.287135 (+ 561us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.287239 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260201 03:02:21.287461 21223 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:02:21.288214 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.276312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35461 (local address 127.18.203.194:45685)
0201 03:02:21.276758 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.276793 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.277542 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.278065 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.278075 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.278586 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:21.279180 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.279192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.280108 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.280111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.280519 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.280526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.280740 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.281381 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:21.281420 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:21.283453 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:21.285491 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.285496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.285506 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.285732 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.286034 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.286038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.286040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.286162 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.286592 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:21.286598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:21.286886 (+ 288us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.287113 (+ 227us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.287387 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":139,"threads_started":1}
I20260201 03:02:21.289234 21354 heartbeater.cc:344] Connected to a master server at 127.18.203.254:35461
I20260201 03:02:21.289388 21354 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:21.289911 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 21223
I20260201 03:02:21.289938 21354 heartbeater.cc:507] Master 127.18.203.254:35461 requested a full tablet report, sending...
I20260201 03:02:21.290019 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:02:21.290299 19247 external_mini_cluster.cc:1468] Setting key a8e2ef49cfc71db00654ecc651a83563
I20260201 03:02:21.290989 21006 ts_manager.cc:194] Registered new tserver with Master: f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659)
I20260201 03:02:21.291633 21006 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:45685
I20260201 03:02:21.298156 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.291290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35461 (local address 127.18.203.194:53957)
0201 03:02:21.291603 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.291639 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.291736 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.291966 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.291969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.292160 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:21.292380 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.292386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.293051 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.293055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.293411 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.293418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.293517 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.294726 (+ 1209us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:21.294739 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:21.295038 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:21.296846 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.296850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.296852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.297062 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.297333 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.297335 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.297337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.297383 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.297755 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:21.297757 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:21.297835 (+ 78us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.297909 (+ 74us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.298009 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":76,"threads_started":1}
I20260201 03:02:21.298269 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.291363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:53957 (local address 127.18.203.254:35461)
0201 03:02:21.291489 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.291492 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.291705 (+ 213us) server_negotiation.cc:408] Connection header received
0201 03:02:21.291811 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.291815 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.291860 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.291928 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:21.292488 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.292955 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.293508 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.293684 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.295155 (+ 1471us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:21.295169 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:21.295171 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:21.295193 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:21.296722 (+ 1529us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.297176 (+ 454us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.297180 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.297181 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.297253 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.297459 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.297462 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.297465 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.297674 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:21.297776 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.297983 (+ 207us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.298109 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:21.336390 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:35461
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:21.441432 21365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:21.441664 21365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:21.441756 21365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:21.444983 21365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:21.445048 21365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:21.445120 21365 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260201 03:02:21.445147 21365 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:02:21.445200 21365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:02:21.449340 21365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.203.254:35461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:21.450387 21365 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:21.451228 21365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:21.457444 21370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:21.457509 21371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:21.457767 21365 server_base.cc:1060] running on GCE node
W20260201 03:02:21.457516 21373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:21.458213 21365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:21.458736 21365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:21.459872 21365 hybrid_clock.cc:648] HybridClock initialized: now 1769914941459858 us; error 35 us; skew 500 ppm
Feb 01 03:02:21 dist-test-slave-gf59 krb5kdc[20959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914941, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:21.462358 21365 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:02:21.463361 21365 webserver.cc:492] Webserver started at http://127.18.203.195:33863/ using document root <none> and password file <none>
I20260201 03:02:21.463933 21365 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:21.463982 21365 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:21.464143 21365 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:21.465812 21365 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "142866a5294545a9b190f6c2f66a9cc6"
format_stamp: "Formatted at 2026-02-01 03:02:21 on dist-test-slave-gf59"
server_key: "d019927cf177776999acc49753d4f608"
server_key_iv: "1689b57da54bbe4108ca302b9f818a36"
server_key_version: "encryptionkey@0"
I20260201 03:02:21.466230 21365 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "142866a5294545a9b190f6c2f66a9cc6"
format_stamp: "Formatted at 2026-02-01 03:02:21 on dist-test-slave-gf59"
server_key: "d019927cf177776999acc49753d4f608"
server_key_iv: "1689b57da54bbe4108ca302b9f818a36"
server_key_version: "encryptionkey@0"
I20260201 03:02:21.469367 21365 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:02:21.471347 21380 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:21.472304 21365 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:21.472391 21365 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "142866a5294545a9b190f6c2f66a9cc6"
format_stamp: "Formatted at 2026-02-01 03:02:21 on dist-test-slave-gf59"
server_key: "d019927cf177776999acc49753d4f608"
server_key_iv: "1689b57da54bbe4108ca302b9f818a36"
server_key_version: "encryptionkey@0"
I20260201 03:02:21.472460 21365 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:21.484948 21365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:21.487632 21365 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:21.487792 21365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:21.489137 21365 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:21.489190 21365 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:21.489243 21365 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:21.489272 21365 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:02:21 dist-test-slave-gf59 krb5kdc[20959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914941, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:21.499259 21365 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:41125
I20260201 03:02:21.499296 21495 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:41125 every 8 connection(s)
I20260201 03:02:21.500402 21365 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:02:21.501299 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 21365
I20260201 03:02:21.501333 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.489755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:38933 (local address 127.18.203.254:35461)
0201 03:02:21.489896 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.489901 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.490670 (+ 769us) server_negotiation.cc:408] Connection header received
0201 03:02:21.491575 (+ 905us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.491579 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.491663 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.491760 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:21.493369 (+ 1609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.494147 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.495039 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.495255 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.497897 (+ 2642us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:21.497919 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:21.497922 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:21.497957 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:21.499432 (+ 1475us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.499946 (+ 514us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.499949 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.499950 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.499991 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.500253 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.500255 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.500256 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.500395 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:21.500473 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.501047 (+ 574us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.501171 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260201 03:02:21.501413 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:02:21.501665 19247 external_mini_cluster.cc:1468] Setting key fa33b856db5d5d43b386eebd79fedc22
I20260201 03:02:21.501968 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.490049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35461 (local address 127.18.203.195:38933)
0201 03:02:21.490527 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.490567 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.491345 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.491932 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.491943 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.492466 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:21.493201 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.493219 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.494280 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.494284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.494897 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.494907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.495130 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.495794 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:21.495820 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:21.497723 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:21.499532 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.499536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.499546 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.499858 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.500070 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.500073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.500075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.500179 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.500486 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:21.500493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:21.500770 (+ 277us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.500989 (+ 219us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.501270 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":118,"threads_started":1}
I20260201 03:02:21.503634 21496 heartbeater.cc:344] Connected to a master server at 127.18.203.254:35461
I20260201 03:02:21.503798 21496 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:21.504300 21496 heartbeater.cc:507] Master 127.18.203.254:35461 requested a full tablet report, sending...
I20260201 03:02:21.505191 21007 ts_manager.cc:194] Registered new tserver with Master: 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125)
I20260201 03:02:21.505942 21007 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:38933
I20260201 03:02:21.510809 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.505495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:51953 (local address 127.18.203.254:35461)
0201 03:02:21.505596 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.505599 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.505802 (+ 203us) server_negotiation.cc:408] Connection header received
0201 03:02:21.505909 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.505912 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.505947 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.506016 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:21.506596 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.507288 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.507859 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.508017 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.508642 (+ 625us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:21.508655 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:21.508657 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:21.508676 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:21.509850 (+ 1174us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.510127 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.510130 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.510131 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.510165 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.510301 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.510303 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.510304 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.510413 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:21.510475 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.510628 (+ 153us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.510704 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260201 03:02:21.510811 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.505454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35461 (local address 127.18.203.195:51953)
0201 03:02:21.505731 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.505744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.505834 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.506045 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.506048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.506238 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:21.506496 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.506503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.507379 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.507381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.507763 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.507770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.507853 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.508288 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:21.508299 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:21.508549 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:21.509925 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.509927 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.509928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.510064 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.510219 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.510221 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.510222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.510256 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.510468 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:21.510470 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:21.510531 (+ 61us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.510599 (+ 68us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.510688 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":76,"threads_started":1}
I20260201 03:02:21.515123 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 01 03:02:21 dist-test-slave-gf59 krb5kdc[20959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 01 03:02:21 dist-test-slave-gf59 krb5kdc[20959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:21.538633 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.531971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.18.203.254:35461)
0201 03:02:21.532096 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.532099 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.532112 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:02:21.532153 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.532156 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.532199 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.532270 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:21.533012 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.533499 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.534161 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.534294 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.536280 (+ 1986us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:21.536299 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:21.536301 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:21.536327 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:21.537532 (+ 1205us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.537806 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.537808 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.537810 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.537847 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.537977 (+ 130us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.537978 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.537980 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.538125 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:21.538216 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.538412 (+ 196us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.538508 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260201 03:02:21.541013 21007 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:02:21.543428 21007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:02:21.555774 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.551204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:41125 (local address 127.0.0.1:44558)
0201 03:02:21.551941 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.551980 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.552116 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.553092 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.553095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.553117 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:21.553361 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.553368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.554403 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.554406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.555279 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.555286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.555381 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.555436 (+ 55us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.555522 (+ 86us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.555629 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":68,"threads_started":1}
I20260201 03:02:21.556288 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.551397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:33429 (local address 127.0.0.1:50228)
0201 03:02:21.552334 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.552346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.552436 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.552907 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.552912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.552933 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:21.553254 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.553264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.554671 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.554674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.555702 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.555710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.555771 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.555781 (+ 10us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.555818 (+ 37us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.556165 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":92,"threads_started":1}
I20260201 03:02:21.556370 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.551325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.18.203.195:41125)
0201 03:02:21.552680 (+ 1355us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.552686 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.552705 (+ 19us) server_negotiation.cc:408] Connection header received
0201 03:02:21.552778 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.552782 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.552936 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.553110 (+ 174us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:21.553492 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.554249 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.555400 (+ 1151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.556030 (+ 630us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.556146 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.556197 (+ 51us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.556262 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1257,"thread_start_us":108,"threads_started":1}
I20260201 03:02:21.556862 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.552199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.18.203.194:41659)
0201 03:02:21.552479 (+ 280us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.552484 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.552683 (+ 199us) server_negotiation.cc:408] Connection header received
0201 03:02:21.552807 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.552812 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.552974 (+ 162us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.553124 (+ 150us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:21.553600 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.554675 (+ 1075us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.555980 (+ 1305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.556463 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.556577 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.556635 (+ 58us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.556717 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":95,"threads_started":1}
I20260201 03:02:21.557168 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.551814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:41659 (local address 127.0.0.1:33000)
0201 03:02:21.552589 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.552600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.552689 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.553275 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.553278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.553293 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:21.553491 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.553498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.554780 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.554784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.555647 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.555657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.555763 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.555781 (+ 18us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.555837 (+ 56us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.557055 (+ 1218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":81,"threads_started":1}
I20260201 03:02:21.559140 21430 tablet_service.cc:1505] Processing CreateTablet for tablet e191f04c4763446f90fd0cfcd172833d (DEFAULT_TABLE table=test-table [id=ac1a67d50938451ba0715d33e2b4cbb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:21.559564 21288 tablet_service.cc:1505] Processing CreateTablet for tablet e191f04c4763446f90fd0cfcd172833d (DEFAULT_TABLE table=test-table [id=ac1a67d50938451ba0715d33e2b4cbb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:21.560066 21430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e191f04c4763446f90fd0cfcd172833d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:21.560379 21288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e191f04c4763446f90fd0cfcd172833d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:21.565248 21520 tablet_bootstrap.cc:492] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6: Bootstrap starting.
I20260201 03:02:21.565248 21519 tablet_bootstrap.cc:492] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e: Bootstrap starting.
I20260201 03:02:21.566792 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.551981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.18.203.193:33429)
0201 03:02:21.552314 (+ 333us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.552321 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.552421 (+ 100us) server_negotiation.cc:408] Connection header received
0201 03:02:21.552554 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.552560 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.552745 (+ 185us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.552897 (+ 152us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:21.553390 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.554523 (+ 1133us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.556618 (+ 2095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.566340 (+ 9722us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.566463 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.566536 (+ 73us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.566640 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":88,"threads_started":1}
I20260201 03:02:21.567535 21519 tablet_bootstrap.cc:654] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:21.567555 21520 tablet_bootstrap.cc:654] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:21.568552 21519 log.cc:826] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:21.568723 21520 log.cc:826] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:21.569700 21146 tablet_service.cc:1505] Processing CreateTablet for tablet e191f04c4763446f90fd0cfcd172833d (DEFAULT_TABLE table=test-table [id=ac1a67d50938451ba0715d33e2b4cbb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:21.570308 21520 tablet_bootstrap.cc:492] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6: No bootstrap required, opened a new log
I20260201 03:02:21.570346 21519 tablet_bootstrap.cc:492] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e: No bootstrap required, opened a new log
I20260201 03:02:21.570480 21520 ts_tablet_manager.cc:1403] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260201 03:02:21.570514 21519 ts_tablet_manager.cc:1403] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260201 03:02:21.570667 21146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e191f04c4763446f90fd0cfcd172833d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:21.573035 21520 raft_consensus.cc:359] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } }
I20260201 03:02:21.573207 21520 raft_consensus.cc:385] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:21.573266 21520 raft_consensus.cc:740] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142866a5294545a9b190f6c2f66a9cc6, State: Initialized, Role: FOLLOWER
I20260201 03:02:21.573400 21519 raft_consensus.cc:359] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } }
I20260201 03:02:21.573601 21519 raft_consensus.cc:385] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:21.573657 21519 raft_consensus.cc:740] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c41a09ad844a85982381d531c0d95e, State: Initialized, Role: FOLLOWER
I20260201 03:02:21.573665 21520 consensus_queue.cc:260] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } }
I20260201 03:02:21.574283 21496 heartbeater.cc:499] Master 127.18.203.254:35461 was elected leader, sending a full tablet report...
I20260201 03:02:21.574465 21520 ts_tablet_manager.cc:1434] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260201 03:02:21.574097 21519 consensus_queue.cc:260] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } }
I20260201 03:02:21.575990 21519 ts_tablet_manager.cc:1434] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260201 03:02:21.576340 21354 heartbeater.cc:499] Master 127.18.203.254:35461 was elected leader, sending a full tablet report...
I20260201 03:02:21.576669 21525 tablet_bootstrap.cc:492] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7: Bootstrap starting.
I20260201 03:02:21.578617 21525 tablet_bootstrap.cc:654] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:21.579398 21525 log.cc:826] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:21.580797 21525 tablet_bootstrap.cc:492] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7: No bootstrap required, opened a new log
I20260201 03:02:21.580940 21525 ts_tablet_manager.cc:1403] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260201 03:02:21.583041 21525 raft_consensus.cc:359] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } }
I20260201 03:02:21.583204 21525 raft_consensus.cc:385] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:21.583240 21525 raft_consensus.cc:740] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cdef52faff47e6bae3b60988f108e7, State: Initialized, Role: FOLLOWER
I20260201 03:02:21.583570 21525 consensus_queue.cc:260] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } }
I20260201 03:02:21.584139 21212 heartbeater.cc:499] Master 127.18.203.254:35461 was elected leader, sending a full tablet report...
I20260201 03:02:21.584331 21525 ts_tablet_manager.cc:1434] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260201 03:02:21.648828 21527 raft_consensus.cc:493] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:02:21.649025 21527 raft_consensus.cc:515] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } }
I20260201 03:02:21.650125 21527 leader_election.cc:290] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125), f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659)
I20260201 03:02:21.653232 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.650352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:41659 (local address 127.18.203.193:37983)
0201 03:02:21.650748 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.650760 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.650883 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.651116 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.651120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.651137 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:21.651379 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.651386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.652186 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.652189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.652760 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.652767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.652870 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.652893 (+ 23us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.652942 (+ 49us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.652999 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":359,"threads_started":1}
I20260201 03:02:21.653250 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.650352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:41125 (local address 127.18.203.193:54811)
0201 03:02:21.650908 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.650920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.651020 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.651259 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.651261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.651277 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:21.651503 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.651510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.652251 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.652254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.652820 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.652828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.652932 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.652950 (+ 18us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.652997 (+ 47us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.653050 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":48,"thread_start_us":175,"threads_started":1}
I20260201 03:02:21.653586 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.650468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:37983 (local address 127.18.203.194:41659)
0201 03:02:21.650599 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.650603 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.650844 (+ 241us) server_negotiation.cc:408] Connection header received
0201 03:02:21.650982 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.650985 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.651030 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.651098 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:21.651492 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.652070 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.652872 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.653323 (+ 451us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.653366 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.653419 (+ 53us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.653477 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260201 03:02:21.653615 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.650471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:54811 (local address 127.18.203.195:41125)
0201 03:02:21.650586 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.650589 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.651002 (+ 413us) server_negotiation.cc:408] Connection header received
0201 03:02:21.651126 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.651130 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.651174 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.651237 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:21.651638 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.652146 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.652931 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.653378 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.653412 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.653460 (+ 48us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.653510 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260201 03:02:21.654218 21308 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e191f04c4763446f90fd0cfcd172833d" candidate_uuid: "f7cdef52faff47e6bae3b60988f108e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c41a09ad844a85982381d531c0d95e" is_pre_election: true
I20260201 03:02:21.654219 21450 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e191f04c4763446f90fd0cfcd172833d" candidate_uuid: "f7cdef52faff47e6bae3b60988f108e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142866a5294545a9b190f6c2f66a9cc6" is_pre_election: true
I20260201 03:02:21.654501 21450 raft_consensus.cc:2468] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7cdef52faff47e6bae3b60988f108e7 in term 0.
I20260201 03:02:21.654501 21308 raft_consensus.cc:2468] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7cdef52faff47e6bae3b60988f108e7 in term 0.
I20260201 03:02:21.654956 21097 leader_election.cc:304] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f3c41a09ad844a85982381d531c0d95e, f7cdef52faff47e6bae3b60988f108e7; no voters:
I20260201 03:02:21.655191 21527 raft_consensus.cc:2804] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:02:21.655265 21527 raft_consensus.cc:493] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:02:21.655313 21527 raft_consensus.cc:3060] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:21.656404 21527 raft_consensus.cc:515] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } }
I20260201 03:02:21.656777 21527 leader_election.cc:290] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [CANDIDATE]: Term 1 election: Requested vote from peers 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125), f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659)
I20260201 03:02:21.657088 21450 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e191f04c4763446f90fd0cfcd172833d" candidate_uuid: "f7cdef52faff47e6bae3b60988f108e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142866a5294545a9b190f6c2f66a9cc6"
I20260201 03:02:21.657115 21308 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e191f04c4763446f90fd0cfcd172833d" candidate_uuid: "f7cdef52faff47e6bae3b60988f108e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c41a09ad844a85982381d531c0d95e"
I20260201 03:02:21.657202 21450 raft_consensus.cc:3060] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:21.657212 21308 raft_consensus.cc:3060] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:21.658123 21308 raft_consensus.cc:2468] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7cdef52faff47e6bae3b60988f108e7 in term 1.
I20260201 03:02:21.658138 21450 raft_consensus.cc:2468] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7cdef52faff47e6bae3b60988f108e7 in term 1.
I20260201 03:02:21.658403 21097 leader_election.cc:304] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f3c41a09ad844a85982381d531c0d95e, f7cdef52faff47e6bae3b60988f108e7; no voters:
I20260201 03:02:21.658543 21527 raft_consensus.cc:2804] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:21.658761 21527 raft_consensus.cc:697] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 1 LEADER]: Becoming Leader. State: Replica: f7cdef52faff47e6bae3b60988f108e7, State: Running, Role: LEADER
I20260201 03:02:21.659111 21527 consensus_queue.cc:237] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } }
I20260201 03:02:21.662144 21007 catalog_manager.cc:5654] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7cdef52faff47e6bae3b60988f108e7 (127.18.203.193). New cstate: current_term: 1 leader_uuid: "f7cdef52faff47e6bae3b60988f108e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } health_report { overall_health: UNKNOWN } } }
Feb 01 03:02:21 dist-test-slave-gf59 krb5kdc[20959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914940, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:21.696565 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.689403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38494 (local address 127.18.203.254:35461)
0201 03:02:21.689530 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.689535 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.689713 (+ 178us) server_negotiation.cc:408] Connection header received
0201 03:02:21.689876 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.689879 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.689923 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.689988 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:21.690791 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.691275 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.691899 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.692056 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.693951 (+ 1895us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:21.693965 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:21.693967 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:21.693988 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:21.695060 (+ 1072us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.695463 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.695466 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.695468 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.695506 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:21.695753 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:21.695757 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:21.695759 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:21.695933 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:21.696022 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.696282 (+ 260us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.696431 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":41}
I20260201 03:02:21.696565 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.689354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35461 (local address 127.0.0.1:38494)
0201 03:02:21.689632 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.689644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.689778 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.690023 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.690025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.690483 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:21.690675 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.690682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.691373 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.691376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.691788 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.691796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.691908 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.692457 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:21.692471 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:21.693843 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:21.695164 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.695172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.695177 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.695378 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.695595 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:21.695597 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:21.695599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:21.695679 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:21.696028 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:21.696035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:21.696132 (+ 97us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.696230 (+ 98us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.696379 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":117,"threads_started":1}
I20260201 03:02:21.700289 21007 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:38494:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260201 03:02:21.700982 21007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:02:21.705523 21146 tablet_service.cc:1505] Processing CreateTablet for tablet 3173607a6bb9430b85ab950292752b80 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9cf37e1e759549f98a913acb2f5278b8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:02:21.706197 21146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3173607a6bb9430b85ab950292752b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:21.705909 21430 tablet_service.cc:1505] Processing CreateTablet for tablet 3173607a6bb9430b85ab950292752b80 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9cf37e1e759549f98a913acb2f5278b8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:02:21.706445 21430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3173607a6bb9430b85ab950292752b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:21.708590 21525 tablet_bootstrap.cc:492] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7: Bootstrap starting.
I20260201 03:02:21.708650 21520 tablet_bootstrap.cc:492] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6: Bootstrap starting.
I20260201 03:02:21.709520 21525 tablet_bootstrap.cc:654] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:21.709215 21288 tablet_service.cc:1505] Processing CreateTablet for tablet 3173607a6bb9430b85ab950292752b80 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9cf37e1e759549f98a913acb2f5278b8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:02:21.709596 21520 tablet_bootstrap.cc:654] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:21.709748 21288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3173607a6bb9430b85ab950292752b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:21.710675 21525 tablet_bootstrap.cc:492] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7: No bootstrap required, opened a new log
I20260201 03:02:21.710752 21520 tablet_bootstrap.cc:492] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6: No bootstrap required, opened a new log
I20260201 03:02:21.710768 21525 ts_tablet_manager.cc:1403] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:21.710835 21520 ts_tablet_manager.cc:1403] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:21.711222 21525 raft_consensus.cc:359] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:21.711222 21520 raft_consensus.cc:359] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:21.711335 21520 raft_consensus.cc:385] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:21.711335 21525 raft_consensus.cc:385] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:21.711369 21520 raft_consensus.cc:740] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142866a5294545a9b190f6c2f66a9cc6, State: Initialized, Role: FOLLOWER
I20260201 03:02:21.711369 21525 raft_consensus.cc:740] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cdef52faff47e6bae3b60988f108e7, State: Initialized, Role: FOLLOWER
I20260201 03:02:21.711488 21520 consensus_queue.cc:260] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:21.711488 21525 consensus_queue.cc:260] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:21.711766 21520 ts_tablet_manager.cc:1434] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:02:21.711814 21525 ts_tablet_manager.cc:1434] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:02:21.711755 21527 tablet_replica.cc:442] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } }
I20260201 03:02:21.711882 21527 tablet_replica.cc:445] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:21.712019 21523 tablet_replica.cc:442] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } }
I20260201 03:02:21.712131 21523 tablet_replica.cc:445] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:21.712168 21519 tablet_bootstrap.cc:492] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: Bootstrap starting.
I20260201 03:02:21.713132 21519 tablet_bootstrap.cc:654] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:21.714768 21519 tablet_bootstrap.cc:492] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: No bootstrap required, opened a new log
I20260201 03:02:21.714852 21519 ts_tablet_manager.cc:1403] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260201 03:02:21.715232 21519 raft_consensus.cc:359] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:21.715330 21519 raft_consensus.cc:385] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:21.715366 21519 raft_consensus.cc:740] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c41a09ad844a85982381d531c0d95e, State: Initialized, Role: FOLLOWER
I20260201 03:02:21.715492 21519 consensus_queue.cc:260] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:21.715739 21519 ts_tablet_manager.cc:1434] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:02:21.716003 21524 tablet_replica.cc:442] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } }
I20260201 03:02:21.716130 21524 tablet_replica.cc:445] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260201 03:02:21.751302 21497 tablet.cc:2378] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260201 03:02:21.788623 21355 tablet.cc:2378] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260201 03:02:21.803753 21213 tablet.cc:2378] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:02:21.888988 21524 raft_consensus.cc:493] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:02:21.889144 21524 raft_consensus.cc:515] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:21.890501 21524 leader_election.cc:290] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7cdef52faff47e6bae3b60988f108e7 (127.18.203.193:33429), 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125)
I20260201 03:02:21.893440 21542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.890745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:41125 (local address 127.18.203.194:48801)
0201 03:02:21.891092 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.891103 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.891207 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.891434 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.891437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.891487 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:21.891738 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.891749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.892430 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.892434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.893026 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.893033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.893138 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.893160 (+ 22us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.893209 (+ 49us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.893272 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"spinlock_wait_cycles":28032,"thread_start_us":101,"threads_started":1}
I20260201 03:02:21.893455 21541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.890730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:33429 (local address 127.18.203.194:42405)
0201 03:02:21.891050 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.891065 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.891197 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.891434 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.891436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.891456 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:21.891712 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.891719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.892442 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.892446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.893019 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.893026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.893141 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.893161 (+ 20us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.893211 (+ 50us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.893269 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"spinlock_wait_cycles":768,"thread_start_us":119,"threads_started":1}
I20260201 03:02:21.893848 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.890878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:48801 (local address 127.18.203.195:41125)
0201 03:02:21.891051 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.891055 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.891181 (+ 126us) server_negotiation.cc:408] Connection header received
0201 03:02:21.891302 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.891305 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.891348 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.891412 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:21.891870 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.892328 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.893145 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.893582 (+ 437us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.893622 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.893670 (+ 48us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.893722 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260201 03:02:21.893855 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.890853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:42405 (local address 127.18.203.193:33429)
0201 03:02:21.890993 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.890997 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.891159 (+ 162us) server_negotiation.cc:408] Connection header received
0201 03:02:21.891298 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.891300 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.891342 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.891405 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:21.891819 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.892333 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.893137 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.893603 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.893642 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.893689 (+ 47us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.893751 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260201 03:02:21.894243 21450 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3173607a6bb9430b85ab950292752b80" candidate_uuid: "f3c41a09ad844a85982381d531c0d95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142866a5294545a9b190f6c2f66a9cc6" is_pre_election: true
I20260201 03:02:21.894407 21450 raft_consensus.cc:2468] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c41a09ad844a85982381d531c0d95e in term 0.
I20260201 03:02:21.894572 21166 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3173607a6bb9430b85ab950292752b80" candidate_uuid: "f3c41a09ad844a85982381d531c0d95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cdef52faff47e6bae3b60988f108e7" is_pre_election: true
I20260201 03:02:21.894778 21166 raft_consensus.cc:2468] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c41a09ad844a85982381d531c0d95e in term 0.
I20260201 03:02:21.894816 21242 leader_election.cc:304] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142866a5294545a9b190f6c2f66a9cc6, f3c41a09ad844a85982381d531c0d95e; no voters:
I20260201 03:02:21.895071 21524 raft_consensus.cc:2804] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:02:21.895133 21524 raft_consensus.cc:493] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:02:21.895179 21524 raft_consensus.cc:3060] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:21.896167 21524 raft_consensus.cc:515] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:21.896486 21524 leader_election.cc:290] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [CANDIDATE]: Term 1 election: Requested vote from peers f7cdef52faff47e6bae3b60988f108e7 (127.18.203.193:33429), 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125)
I20260201 03:02:21.896836 21450 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3173607a6bb9430b85ab950292752b80" candidate_uuid: "f3c41a09ad844a85982381d531c0d95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142866a5294545a9b190f6c2f66a9cc6"
I20260201 03:02:21.896828 21166 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3173607a6bb9430b85ab950292752b80" candidate_uuid: "f3c41a09ad844a85982381d531c0d95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cdef52faff47e6bae3b60988f108e7"
I20260201 03:02:21.896942 21450 raft_consensus.cc:3060] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:21.896942 21166 raft_consensus.cc:3060] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:21.897980 21450 raft_consensus.cc:2468] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c41a09ad844a85982381d531c0d95e in term 1.
I20260201 03:02:21.898244 21242 leader_election.cc:304] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142866a5294545a9b190f6c2f66a9cc6, f3c41a09ad844a85982381d531c0d95e; no voters:
I20260201 03:02:21.898437 21524 raft_consensus.cc:2804] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:21.898438 21166 raft_consensus.cc:2468] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c41a09ad844a85982381d531c0d95e in term 1.
I20260201 03:02:21.898607 21524 raft_consensus.cc:697] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [term 1 LEADER]: Becoming Leader. State: Replica: f3c41a09ad844a85982381d531c0d95e, State: Running, Role: LEADER
I20260201 03:02:21.898927 21524 consensus_queue.cc:237] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:21.900902 21543 tablet_replica.cc:442] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: TxnStatusTablet state changed. Reason: New leader f3c41a09ad844a85982381d531c0d95e. Latest consensus state: current_term: 1 leader_uuid: "f3c41a09ad844a85982381d531c0d95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } }
I20260201 03:02:21.901005 21543 tablet_replica.cc:445] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:02:21.901273 21545 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260201 03:02:21.901454 21545 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260201 03:02:21.901904 21005 catalog_manager.cc:5654] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c41a09ad844a85982381d531c0d95e (127.18.203.194). New cstate: current_term: 1 leader_uuid: "f3c41a09ad844a85982381d531c0d95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } health_report { overall_health: UNKNOWN } } }
I20260201 03:02:21.949880 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.946467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:41659 (local address 127.0.0.1:33012)
0201 03:02:21.946626 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.946641 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.946746 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.947077 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.947079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.947086 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:02:21.947271 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.947277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.948250 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.948252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.948651 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.948657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.948740 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.948780 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:02:21.949646 (+ 866us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:02:21.949655 (+ 9us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.949687 (+ 32us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.949749 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260201 03:02:21.949913 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.946579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.18.203.194:41659)
0201 03:02:21.946728 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.946733 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.946750 (+ 17us) server_negotiation.cc:408] Connection header received
0201 03:02:21.946903 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.946906 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.946979 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.947077 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:21.947366 (+ 289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.948155 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.948749 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.948948 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.949006 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:21.949563 (+ 557us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:21.949677 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.949714 (+ 37us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.949770 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260201 03:02:21.954947 21166 raft_consensus.cc:1275] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Refusing update from remote peer f3c41a09ad844a85982381d531c0d95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:21.955063 21450 raft_consensus.cc:1275] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Refusing update from remote peer f3c41a09ad844a85982381d531c0d95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:21.955541 21543 consensus_queue.cc:1048] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:21.955744 21524 consensus_queue.cc:1048] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:21.958683 21527 tablet_replica.cc:442] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7: TxnStatusTablet state changed. Reason: New leader f3c41a09ad844a85982381d531c0d95e. Latest consensus state: current_term: 1 leader_uuid: "f3c41a09ad844a85982381d531c0d95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } }
I20260201 03:02:21.958794 21527 tablet_replica.cc:445] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:21.959381 21523 tablet_replica.cc:442] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6: TxnStatusTablet state changed. Reason: New leader f3c41a09ad844a85982381d531c0d95e. Latest consensus state: current_term: 1 leader_uuid: "f3c41a09ad844a85982381d531c0d95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } }
I20260201 03:02:21.959488 21523 tablet_replica.cc:445] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:21.963119 21543 tablet_replica.cc:442] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3c41a09ad844a85982381d531c0d95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } }
I20260201 03:02:21.963229 21543 tablet_replica.cc:445] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:02:21.963429 21527 tablet_replica.cc:442] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3c41a09ad844a85982381d531c0d95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } }
I20260201 03:02:21.963527 21527 tablet_replica.cc:445] T 3173607a6bb9430b85ab950292752b80 P f7cdef52faff47e6bae3b60988f108e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:21.963536 21543 tablet_replica.cc:442] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3c41a09ad844a85982381d531c0d95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } }
I20260201 03:02:21.963639 21543 tablet_replica.cc:445] T 3173607a6bb9430b85ab950292752b80 P f3c41a09ad844a85982381d531c0d95e: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:02:21.964027 21548 mvcc.cc:204] Tried to move back new op lower bound from 7249571602241138688 to 7249571602020474880. Current Snapshot: MvccSnapshot[applied={T|T < 7249571602241138688}]
I20260201 03:02:21.964818 21523 tablet_replica.cc:442] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3c41a09ad844a85982381d531c0d95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } } }
I20260201 03:02:21.964922 21523 tablet_replica.cc:445] T 3173607a6bb9430b85ab950292752b80 P 142866a5294545a9b190f6c2f66a9cc6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:02:21.965380 21550 mvcc.cc:204] Tried to move back new op lower bound from 7249571602241138688 to 7249571602020474880. Current Snapshot: MvccSnapshot[applied={T|T < 7249571602241138688}]
I20260201 03:02:21.975515 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.972681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.18.203.193:33429)
0201 03:02:21.972792 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.972797 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.972810 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:02:21.972952 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.972955 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.973009 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.973095 (+ 86us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:21.973418 (+ 323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.973877 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.974495 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.974658 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.974709 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:21.975156 (+ 447us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:21.975257 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.975337 (+ 80us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.975388 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260201 03:02:21.985551 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.982396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:41659 (local address 127.18.203.193:59085)
0201 03:02:21.982786 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:21.982798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:21.982891 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:21.983126 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:21.983128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:21.983135 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:02:21.983361 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.983366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.984323 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.984326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:21.984743 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:21.984749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.984870 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.984916 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:02:21.985260 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:02:21.985265 (+ 5us) client_negotiation.cc:769] Sending connection context
0201 03:02:21.985321 (+ 56us) client_negotiation.cc:240] Negotiation successful
0201 03:02:21.985386 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":105,"threads_started":1}
I20260201 03:02:21.985522 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:21.982483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:59085 (local address 127.18.203.194:41659)
0201 03:02:21.982633 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:21.982638 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:21.982871 (+ 233us) server_negotiation.cc:408] Connection header received
0201 03:02:21.982977 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:21.982980 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:21.983038 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:21.983116 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:21.983449 (+ 333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.984227 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:21.984867 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:21.985063 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:21.985107 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:21.985176 (+ 69us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:21.985239 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:21.985338 (+ 99us) server_negotiation.cc:300] Negotiation successful
0201 03:02:21.985382 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260201 03:02:21.992086 21450 raft_consensus.cc:1275] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Refusing update from remote peer f7cdef52faff47e6bae3b60988f108e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:21.992379 21308 raft_consensus.cc:1275] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Refusing update from remote peer f7cdef52faff47e6bae3b60988f108e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:21.992875 21530 consensus_queue.cc:1048] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:21.993033 21527 consensus_queue.cc:1048] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:22.016167 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:22.012780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:52159 (local address 127.18.203.193:33429)
0201 03:02:22.012902 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:22.012906 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:22.013124 (+ 218us) server_negotiation.cc:408] Connection header received
0201 03:02:22.013288 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:22.013292 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:22.013337 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:22.013396 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:22.013999 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.014568 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:22.015582 (+ 1014us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.015754 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:22.015796 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:22.015870 (+ 74us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:22.015926 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:22.016019 (+ 93us) server_negotiation.cc:300] Negotiation successful
0201 03:02:22.016059 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260201 03:02:22.016237 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:22.012721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:33429 (local address 127.18.203.194:52159)
0201 03:02:22.013034 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:22.013049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:22.013196 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:22.013412 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:22.013415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:22.013423 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:02:22.013656 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:22.013662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.014816 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:22.014820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:22.015475 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:22.015485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.015600 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:22.015682 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:02:22.015946 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:02:22.015951 (+ 5us) client_negotiation.cc:769] Sending connection context
0201 03:02:22.016014 (+ 63us) client_negotiation.cc:240] Negotiation successful
0201 03:02:22.016093 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":107,"threads_started":1}
I20260201 03:02:22.041922 21007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:22.042124 21007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:22.044924 21007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b: Sending DeleteTablet for 3 replicas of tablet e191f04c4763446f90fd0cfcd172833d
I20260201 03:02:22.045635 21430 tablet_service.cc:1552] Processing DeleteTablet for tablet e191f04c4763446f90fd0cfcd172833d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:22 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:44558
I20260201 03:02:22.045923 21288 tablet_service.cc:1552] Processing DeleteTablet for tablet e191f04c4763446f90fd0cfcd172833d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:22 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:33000
I20260201 03:02:22.046275 21568 tablet_replica.cc:333] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6: stopping tablet replica
I20260201 03:02:22.046546 21568 raft_consensus.cc:2243] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:22.046761 21568 raft_consensus.cc:2272] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:22.047245 21146 tablet_service.cc:1552] Processing DeleteTablet for tablet e191f04c4763446f90fd0cfcd172833d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:22 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:50228
I20260201 03:02:22.047397 21007 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260201 03:02:22.047724 21570 tablet_replica.cc:333] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7: stopping tablet replica
W20260201 03:02:22.047901 21007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:02:22.047986 21570 raft_consensus.cc:2243] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:02:22.048355 21570 raft_consensus.cc:2272] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:22.049760 21570 ts_tablet_manager.cc:1916] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:22.053689 21146 tablet_service.cc:1505] Processing CreateTablet for tablet a89470aee5cd4475b0ad17160409aaa5 (DEFAULT_TABLE table=test-table [id=455f1e2f589944e5a57039f23b049dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:22.054010 21146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89470aee5cd4475b0ad17160409aaa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:22.054126 21569 tablet_replica.cc:333] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e: stopping tablet replica
I20260201 03:02:22.054416 21569 raft_consensus.cc:2243] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:22.054639 21569 raft_consensus.cc:2272] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:22.055970 21569 ts_tablet_manager.cc:1916] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:22.056700 21288 tablet_service.cc:1505] Processing CreateTablet for tablet a89470aee5cd4475b0ad17160409aaa5 (DEFAULT_TABLE table=test-table [id=455f1e2f589944e5a57039f23b049dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:22.056978 21288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89470aee5cd4475b0ad17160409aaa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:22.058997 21569 ts_tablet_manager.cc:1929] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:02:22.059653 21525 tablet_bootstrap.cc:492] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7: Bootstrap starting.
I20260201 03:02:22.060616 21525 tablet_bootstrap.cc:654] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:22.062445 21430 tablet_service.cc:1505] Processing CreateTablet for tablet a89470aee5cd4475b0ad17160409aaa5 (DEFAULT_TABLE table=test-table [id=455f1e2f589944e5a57039f23b049dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:22.059182 21519 tablet_bootstrap.cc:492] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e: Bootstrap starting.
I20260201 03:02:22.062768 21430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89470aee5cd4475b0ad17160409aaa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:22.063596 21519 tablet_bootstrap.cc:654] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:22.064670 21569 log.cc:1199] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/e191f04c4763446f90fd0cfcd172833d
I20260201 03:02:22.065049 21569 ts_tablet_manager.cc:1950] T e191f04c4763446f90fd0cfcd172833d P f3c41a09ad844a85982381d531c0d95e: Deleting consensus metadata
I20260201 03:02:22.065093 21520 tablet_bootstrap.cc:492] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6: Bootstrap starting.
I20260201 03:02:22.065590 21570 ts_tablet_manager.cc:1929] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:02:22.065685 21570 log.cc:1199] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/wal/wals/e191f04c4763446f90fd0cfcd172833d
I20260201 03:02:22.065985 21570 ts_tablet_manager.cc:1950] T e191f04c4763446f90fd0cfcd172833d P f7cdef52faff47e6bae3b60988f108e7: Deleting consensus metadata
I20260201 03:02:22.066021 21520 tablet_bootstrap.cc:654] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:22.066606 20991 catalog_manager.cc:4985] TS f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659): tablet e191f04c4763446f90fd0cfcd172833d (table test-table [id=ac1a67d50938451ba0715d33e2b4cbb0]) successfully deleted
I20260201 03:02:22.067020 20992 catalog_manager.cc:4985] TS f7cdef52faff47e6bae3b60988f108e7 (127.18.203.193:33429): tablet e191f04c4763446f90fd0cfcd172833d (table test-table [id=ac1a67d50938451ba0715d33e2b4cbb0]) successfully deleted
I20260201 03:02:22.067797 21568 ts_tablet_manager.cc:1916] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:22.071784 21525 tablet_bootstrap.cc:492] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7: No bootstrap required, opened a new log
I20260201 03:02:22.071898 21525 ts_tablet_manager.cc:1403] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260201 03:02:22.072405 21525 raft_consensus.cc:359] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:22.072554 21525 raft_consensus.cc:385] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:22.072592 21525 raft_consensus.cc:740] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cdef52faff47e6bae3b60988f108e7, State: Initialized, Role: FOLLOWER
I20260201 03:02:22.072646 21519 tablet_bootstrap.cc:492] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e: No bootstrap required, opened a new log
I20260201 03:02:22.072738 21519 ts_tablet_manager.cc:1403] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20260201 03:02:22.072721 21525 consensus_queue.cc:260] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:22.073045 21525 ts_tablet_manager.cc:1434] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:02:22.073176 21519 raft_consensus.cc:359] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:22.073274 21519 raft_consensus.cc:385] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:22.073303 21519 raft_consensus.cc:740] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c41a09ad844a85982381d531c0d95e, State: Initialized, Role: FOLLOWER
I20260201 03:02:22.073414 21519 consensus_queue.cc:260] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:22.073639 21519 ts_tablet_manager.cc:1434] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260201 03:02:22.075832 21568 ts_tablet_manager.cc:1929] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:02:22.075937 21568 log.cc:1199] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/e191f04c4763446f90fd0cfcd172833d
I20260201 03:02:22.076313 21568 ts_tablet_manager.cc:1950] T e191f04c4763446f90fd0cfcd172833d P 142866a5294545a9b190f6c2f66a9cc6: Deleting consensus metadata
I20260201 03:02:22.078339 20994 catalog_manager.cc:4985] TS 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125): tablet e191f04c4763446f90fd0cfcd172833d (table test-table [id=ac1a67d50938451ba0715d33e2b4cbb0]) successfully deleted
I20260201 03:02:22.080382 21520 tablet_bootstrap.cc:492] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6: No bootstrap required, opened a new log
I20260201 03:02:22.080475 21520 ts_tablet_manager.cc:1403] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260201 03:02:22.080940 21520 raft_consensus.cc:359] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:22.081046 21520 raft_consensus.cc:385] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:22.081079 21520 raft_consensus.cc:740] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142866a5294545a9b190f6c2f66a9cc6, State: Initialized, Role: FOLLOWER
I20260201 03:02:22.081184 21520 consensus_queue.cc:260] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:22.081411 21520 ts_tablet_manager.cc:1434] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:02:22.093890 21523 raft_consensus.cc:493] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:02:22.094026 21523 raft_consensus.cc:515] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:22.095203 21523 leader_election.cc:290] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7cdef52faff47e6bae3b60988f108e7 (127.18.203.193:33429), f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659)
I20260201 03:02:22.100004 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:22.095799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:39835 (local address 127.18.203.194:41659)
0201 03:02:22.095930 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:22.095934 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:22.096136 (+ 202us) server_negotiation.cc:408] Connection header received
0201 03:02:22.096313 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:22.096317 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:22.096363 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:22.096408 (+ 45us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:22.096988 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.097722 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:22.098924 (+ 1202us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.099395 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:22.099434 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:22.099801 (+ 367us) server_negotiation.cc:300] Negotiation successful
0201 03:02:22.099878 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260201 03:02:22.100358 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:22.095456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:41659 (local address 127.18.203.195:39835)
0201 03:02:22.096057 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:22.096070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:22.096205 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:22.096566 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:22.096569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:22.096585 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:22.096873 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:22.096879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.097855 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:22.097858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:22.098790 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:22.098798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.099654 (+ 856us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:22.099673 (+ 19us) client_negotiation.cc:769] Sending connection context
0201 03:02:22.100170 (+ 497us) client_negotiation.cc:240] Negotiation successful
0201 03:02:22.100230 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":87,"threads_started":1}
I20260201 03:02:22.100723 21308 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a89470aee5cd4475b0ad17160409aaa5" candidate_uuid: "142866a5294545a9b190f6c2f66a9cc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c41a09ad844a85982381d531c0d95e" is_pre_election: true
I20260201 03:02:22.100867 21308 raft_consensus.cc:2468] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142866a5294545a9b190f6c2f66a9cc6 in term 0.
I20260201 03:02:22.100951 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:22.095798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:41749 (local address 127.18.203.193:33429)
0201 03:02:22.095929 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:22.095934 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:22.096295 (+ 361us) server_negotiation.cc:408] Connection header received
0201 03:02:22.096474 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:22.096477 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:22.096526 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:22.096571 (+ 45us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:22.097159 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.097891 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:22.099087 (+ 1196us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.099561 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:22.099597 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:22.100754 (+ 1157us) server_negotiation.cc:300] Negotiation successful
0201 03:02:22.100836 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260201 03:02:22.101195 21381 leader_election.cc:304] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142866a5294545a9b190f6c2f66a9cc6, f3c41a09ad844a85982381d531c0d95e; no voters:
I20260201 03:02:22.101409 21523 raft_consensus.cc:2804] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:02:22.101461 21523 raft_consensus.cc:493] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:02:22.101500 21523 raft_consensus.cc:3060] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:22.101706 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:22.095484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:33429 (local address 127.18.203.195:41749)
0201 03:02:22.096216 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:22.096230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:22.096362 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:22.096688 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:22.096691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:22.096707 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:22.097025 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:22.097034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.098026 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:22.098029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:22.098952 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:22.098961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.100607 (+ 1646us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:22.100624 (+ 17us) client_negotiation.cc:769] Sending connection context
0201 03:02:22.101408 (+ 784us) client_negotiation.cc:240] Negotiation successful
0201 03:02:22.101573 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":57,"threads_started":1}
I20260201 03:02:22.102202 21166 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a89470aee5cd4475b0ad17160409aaa5" candidate_uuid: "142866a5294545a9b190f6c2f66a9cc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cdef52faff47e6bae3b60988f108e7" is_pre_election: true
I20260201 03:02:22.102327 21166 raft_consensus.cc:2468] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142866a5294545a9b190f6c2f66a9cc6 in term 0.
I20260201 03:02:22.102643 21523 raft_consensus.cc:515] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:22.103009 21523 leader_election.cc:290] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [CANDIDATE]: Term 1 election: Requested vote from peers f7cdef52faff47e6bae3b60988f108e7 (127.18.203.193:33429), f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659)
I20260201 03:02:22.103304 21166 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a89470aee5cd4475b0ad17160409aaa5" candidate_uuid: "142866a5294545a9b190f6c2f66a9cc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cdef52faff47e6bae3b60988f108e7"
I20260201 03:02:22.103406 21166 raft_consensus.cc:3060] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:22.103384 21308 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a89470aee5cd4475b0ad17160409aaa5" candidate_uuid: "142866a5294545a9b190f6c2f66a9cc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c41a09ad844a85982381d531c0d95e"
I20260201 03:02:22.103485 21308 raft_consensus.cc:3060] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:22.104471 21308 raft_consensus.cc:2468] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142866a5294545a9b190f6c2f66a9cc6 in term 1.
I20260201 03:02:22.104576 21166 raft_consensus.cc:2468] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142866a5294545a9b190f6c2f66a9cc6 in term 1.
I20260201 03:02:22.104756 21381 leader_election.cc:304] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142866a5294545a9b190f6c2f66a9cc6, f3c41a09ad844a85982381d531c0d95e; no voters:
I20260201 03:02:22.104915 21523 raft_consensus.cc:2804] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:22.105052 21523 raft_consensus.cc:697] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 LEADER]: Becoming Leader. State: Replica: 142866a5294545a9b190f6c2f66a9cc6, State: Running, Role: LEADER
I20260201 03:02:22.105274 21523 consensus_queue.cc:237] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:22.107600 21007 catalog_manager.cc:5654] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195). New cstate: current_term: 1 leader_uuid: "142866a5294545a9b190f6c2f66a9cc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } health_report { overall_health: HEALTHY } } }
I20260201 03:02:22.130671 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:22.127403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.18.203.195:41125)
0201 03:02:22.127528 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:22.127533 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:22.127548 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:02:22.127697 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:22.127701 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:22.127763 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:22.127873 (+ 110us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:22.128147 (+ 274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.128885 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:22.129432 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.129619 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:22.129678 (+ 59us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:22.130275 (+ 597us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:22.130390 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:22.130463 (+ 73us) server_negotiation.cc:300] Negotiation successful
0201 03:02:22.130536 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260201 03:02:22.140069 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:22.137451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:55157 (local address 127.18.203.194:41659)
0201 03:02:22.137567 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:22.137572 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:22.137727 (+ 155us) server_negotiation.cc:408] Connection header received
0201 03:02:22.137848 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:22.137850 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:22.137895 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:22.137958 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:22.138336 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.138773 (+ 437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:22.139409 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.139585 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:22.139660 (+ 75us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:22.139727 (+ 67us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:22.139788 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:22.139890 (+ 102us) server_negotiation.cc:300] Negotiation successful
0201 03:02:22.139935 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260201 03:02:22.140093 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:22.137383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:41659 (local address 127.18.203.195:55157)
0201 03:02:22.137647 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:22.137659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:22.137754 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:22.137974 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:22.137977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:22.137986 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:02:22.138227 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:22.138233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.138867 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:22.138871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:22.139298 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:22.139305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.139419 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:22.139460 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:02:22.139804 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:02:22.139811 (+ 7us) client_negotiation.cc:769] Sending connection context
0201 03:02:22.139869 (+ 58us) client_negotiation.cc:240] Negotiation successful
0201 03:02:22.139935 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":102,"threads_started":1}
I20260201 03:02:22.145720 21166 raft_consensus.cc:1275] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Refusing update from remote peer 142866a5294545a9b190f6c2f66a9cc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:22.145709 21308 raft_consensus.cc:1275] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Refusing update from remote peer 142866a5294545a9b190f6c2f66a9cc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:22.146155 21576 consensus_queue.cc:1048] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:22.146322 21523 consensus_queue.cc:1048] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:22.163362 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:22.160531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:41125 (local address 127.18.203.194:34559)
0201 03:02:22.160702 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:22.160717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:22.160784 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:22.161067 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:22.161070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:22.161080 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:02:22.161261 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:22.161266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.162218 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:22.162222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:22.162633 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:22.162640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.162737 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:22.162766 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:02:22.163123 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:02:22.163129 (+ 6us) client_negotiation.cc:769] Sending connection context
0201 03:02:22.163183 (+ 54us) client_negotiation.cc:240] Negotiation successful
0201 03:02:22.163251 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260201 03:02:22.163779 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:22.160739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:34559 (local address 127.18.203.195:41125)
0201 03:02:22.160854 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:22.160858 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:22.160871 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:02:22.160911 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:22.160914 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:22.160962 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:22.161032 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:22.161378 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.162116 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:22.162732 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:22.162899 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:22.162937 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:22.163022 (+ 85us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:22.163084 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:22.163594 (+ 510us) server_negotiation.cc:300] Negotiation successful
0201 03:02:22.163672 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260201 03:02:22.164347 21549 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260201 03:02:23.728622 21007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:23.728775 21007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:23.730913 21007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b: Sending DeleteTablet for 3 replicas of tablet a89470aee5cd4475b0ad17160409aaa5
I20260201 03:02:23.731442 21146 tablet_service.cc:1552] Processing DeleteTablet for tablet a89470aee5cd4475b0ad17160409aaa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:23 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:50228
I20260201 03:02:23.731455 21288 tablet_service.cc:1552] Processing DeleteTablet for tablet a89470aee5cd4475b0ad17160409aaa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:23 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:33000
I20260201 03:02:23.731757 21430 tablet_service.cc:1552] Processing DeleteTablet for tablet a89470aee5cd4475b0ad17160409aaa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:23 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:44558
I20260201 03:02:23.731815 21612 tablet_replica.cc:333] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7: stopping tablet replica
I20260201 03:02:23.731822 21611 tablet_replica.cc:333] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e: stopping tablet replica
I20260201 03:02:23.732053 21611 raft_consensus.cc:2243] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:23.732151 21613 tablet_replica.cc:333] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6: stopping tablet replica
I20260201 03:02:23.732249 21611 raft_consensus.cc:2272] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:23.732303 21613 raft_consensus.cc:2243] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:02:23.732399 21612 raft_consensus.cc:2243] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:23.732546 21612 raft_consensus.cc:2272] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:23.732622 21613 raft_consensus.cc:2272] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:23.733218 21612 ts_tablet_manager.cc:1916] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:23.733271 21611 ts_tablet_manager.cc:1916] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:23.733256 21613 ts_tablet_manager.cc:1916] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:23.735669 21613 ts_tablet_manager.cc:1929] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260201 03:02:23.735689 21611 ts_tablet_manager.cc:1929] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260201 03:02:23.735689 21612 ts_tablet_manager.cc:1929] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260201 03:02:23.735742 21613 log.cc:1199] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/a89470aee5cd4475b0ad17160409aaa5
I20260201 03:02:23.735754 21612 log.cc:1199] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/wal/wals/a89470aee5cd4475b0ad17160409aaa5
I20260201 03:02:23.735754 21611 log.cc:1199] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/a89470aee5cd4475b0ad17160409aaa5
I20260201 03:02:23.736016 21613 ts_tablet_manager.cc:1950] T a89470aee5cd4475b0ad17160409aaa5 P 142866a5294545a9b190f6c2f66a9cc6: Deleting consensus metadata
I20260201 03:02:23.736042 21611 ts_tablet_manager.cc:1950] T a89470aee5cd4475b0ad17160409aaa5 P f3c41a09ad844a85982381d531c0d95e: Deleting consensus metadata
I20260201 03:02:23.736042 21612 ts_tablet_manager.cc:1950] T a89470aee5cd4475b0ad17160409aaa5 P f7cdef52faff47e6bae3b60988f108e7: Deleting consensus metadata
I20260201 03:02:23.736642 20991 catalog_manager.cc:4985] TS f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659): tablet a89470aee5cd4475b0ad17160409aaa5 (table test-table [id=455f1e2f589944e5a57039f23b049dc9]) successfully deleted
I20260201 03:02:23.736642 20994 catalog_manager.cc:4985] TS 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125): tablet a89470aee5cd4475b0ad17160409aaa5 (table test-table [id=455f1e2f589944e5a57039f23b049dc9]) successfully deleted
I20260201 03:02:23.736866 20992 catalog_manager.cc:4985] TS f7cdef52faff47e6bae3b60988f108e7 (127.18.203.193:33429): tablet a89470aee5cd4475b0ad17160409aaa5 (table test-table [id=455f1e2f589944e5a57039f23b049dc9]) successfully deleted
Feb 01 03:02:23 dist-test-slave-gf59 krb5kdc[20959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914943, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 01 03:02:23 dist-test-slave-gf59 krb5kdc[20959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914943, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:23.758085 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:23.750783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.18.203.254:35461)
0201 03:02:23.751089 (+ 306us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:23.751093 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:23.751108 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:02:23.751151 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:23.751154 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:23.751199 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:23.751279 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:23.752092 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:23.752623 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:23.753191 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:23.753318 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:23.755556 (+ 2238us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:23.755578 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:23.755581 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:23.755629 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:23.756997 (+ 1368us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:23.757319 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:23.757321 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:23.757323 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:23.757359 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:23.757493 (+ 134us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:23.757495 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:23.757496 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:23.757627 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:23.757713 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:23.757840 (+ 127us) server_negotiation.cc:300] Negotiation successful
0201 03:02:23.757938 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":119,"threads_started":1}
I20260201 03:02:23.760285 21007 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:02:23.760689 21007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:02:23.764972 21146 tablet_service.cc:1505] Processing CreateTablet for tablet ea9b844f1ae747bfb5468d7569aa0d85 (DEFAULT_TABLE table=test-table [id=607810f364a446d6866cc6cd52761305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:23.765141 21288 tablet_service.cc:1505] Processing CreateTablet for tablet ea9b844f1ae747bfb5468d7569aa0d85 (DEFAULT_TABLE table=test-table [id=607810f364a446d6866cc6cd52761305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:23.765277 21146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea9b844f1ae747bfb5468d7569aa0d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:23.765256 21430 tablet_service.cc:1505] Processing CreateTablet for tablet ea9b844f1ae747bfb5468d7569aa0d85 (DEFAULT_TABLE table=test-table [id=607810f364a446d6866cc6cd52761305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:23.765408 21288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea9b844f1ae747bfb5468d7569aa0d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:23.765493 21430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea9b844f1ae747bfb5468d7569aa0d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:23.767557 21622 tablet_bootstrap.cc:492] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e: Bootstrap starting.
I20260201 03:02:23.767844 21624 tablet_bootstrap.cc:492] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7: Bootstrap starting.
I20260201 03:02:23.768584 21622 tablet_bootstrap.cc:654] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:23.768589 21624 tablet_bootstrap.cc:654] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:23.769143 21623 tablet_bootstrap.cc:492] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6: Bootstrap starting.
I20260201 03:02:23.769694 21624 tablet_bootstrap.cc:492] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7: No bootstrap required, opened a new log
I20260201 03:02:23.769713 21622 tablet_bootstrap.cc:492] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e: No bootstrap required, opened a new log
I20260201 03:02:23.769788 21622 ts_tablet_manager.cc:1403] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260201 03:02:23.769788 21624 ts_tablet_manager.cc:1403] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:23.769944 21623 tablet_bootstrap.cc:654] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:23.770192 21622 raft_consensus.cc:359] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:23.770192 21624 raft_consensus.cc:359] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:23.770304 21624 raft_consensus.cc:385] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:23.770358 21624 raft_consensus.cc:740] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cdef52faff47e6bae3b60988f108e7, State: Initialized, Role: FOLLOWER
I20260201 03:02:23.770392 21622 raft_consensus.cc:385] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:23.770473 21622 raft_consensus.cc:740] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c41a09ad844a85982381d531c0d95e, State: Initialized, Role: FOLLOWER
I20260201 03:02:23.770563 21624 consensus_queue.cc:260] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:23.770659 21622 consensus_queue.cc:260] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:23.770937 21622 ts_tablet_manager.cc:1434] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:02:23.771039 21624 ts_tablet_manager.cc:1434] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:02:23.771114 21623 tablet_bootstrap.cc:492] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6: No bootstrap required, opened a new log
I20260201 03:02:23.771198 21623 ts_tablet_manager.cc:1403] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:23.771536 21623 raft_consensus.cc:359] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:23.771667 21623 raft_consensus.cc:385] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:23.771701 21623 raft_consensus.cc:740] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142866a5294545a9b190f6c2f66a9cc6, State: Initialized, Role: FOLLOWER
I20260201 03:02:23.771823 21623 consensus_queue.cc:260] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:23.772075 21623 ts_tablet_manager.cc:1434] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:02:24.047514 21628 raft_consensus.cc:493] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:02:24.047680 21628 raft_consensus.cc:515] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:24.048159 21628 leader_election.cc:290] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659), 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125)
I20260201 03:02:24.048625 21450 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea9b844f1ae747bfb5468d7569aa0d85" candidate_uuid: "f7cdef52faff47e6bae3b60988f108e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142866a5294545a9b190f6c2f66a9cc6" is_pre_election: true
I20260201 03:02:24.048626 21308 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea9b844f1ae747bfb5468d7569aa0d85" candidate_uuid: "f7cdef52faff47e6bae3b60988f108e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c41a09ad844a85982381d531c0d95e" is_pre_election: true
I20260201 03:02:24.048765 21308 raft_consensus.cc:2468] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7cdef52faff47e6bae3b60988f108e7 in term 0.
I20260201 03:02:24.048772 21450 raft_consensus.cc:2468] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7cdef52faff47e6bae3b60988f108e7 in term 0.
I20260201 03:02:24.049113 21097 leader_election.cc:304] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f3c41a09ad844a85982381d531c0d95e, f7cdef52faff47e6bae3b60988f108e7; no voters:
I20260201 03:02:24.049316 21628 raft_consensus.cc:2804] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:02:24.049376 21628 raft_consensus.cc:493] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:02:24.049412 21628 raft_consensus.cc:3060] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:24.050374 21628 raft_consensus.cc:515] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:24.050730 21628 leader_election.cc:290] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [CANDIDATE]: Term 1 election: Requested vote from peers f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659), 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125)
I20260201 03:02:24.051086 21450 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea9b844f1ae747bfb5468d7569aa0d85" candidate_uuid: "f7cdef52faff47e6bae3b60988f108e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142866a5294545a9b190f6c2f66a9cc6"
I20260201 03:02:24.051084 21308 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea9b844f1ae747bfb5468d7569aa0d85" candidate_uuid: "f7cdef52faff47e6bae3b60988f108e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c41a09ad844a85982381d531c0d95e"
I20260201 03:02:24.051194 21308 raft_consensus.cc:3060] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:24.051192 21450 raft_consensus.cc:3060] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:24.052121 21450 raft_consensus.cc:2468] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7cdef52faff47e6bae3b60988f108e7 in term 1.
I20260201 03:02:24.052297 21308 raft_consensus.cc:2468] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7cdef52faff47e6bae3b60988f108e7 in term 1.
I20260201 03:02:24.052397 21100 leader_election.cc:304] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142866a5294545a9b190f6c2f66a9cc6, f7cdef52faff47e6bae3b60988f108e7; no voters:
I20260201 03:02:24.052556 21628 raft_consensus.cc:2804] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:24.052992 21628 raft_consensus.cc:697] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 1 LEADER]: Becoming Leader. State: Replica: f7cdef52faff47e6bae3b60988f108e7, State: Running, Role: LEADER
I20260201 03:02:24.053151 21628 consensus_queue.cc:237] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:24.054926 21005 catalog_manager.cc:5654] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7cdef52faff47e6bae3b60988f108e7 (127.18.203.193). New cstate: current_term: 1 leader_uuid: "f7cdef52faff47e6bae3b60988f108e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } health_report { overall_health: UNKNOWN } } }
I20260201 03:02:24.135154 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:24.132587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.18.203.193:33429)
0201 03:02:24.132846 (+ 259us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:24.132849 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:24.132859 (+ 10us) server_negotiation.cc:408] Connection header received
0201 03:02:24.132893 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:24.132895 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:24.132936 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:24.133008 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:24.133358 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:24.133884 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:24.134507 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:24.134680 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:24.134721 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:24.134786 (+ 65us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:24.134851 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:24.134959 (+ 108us) server_negotiation.cc:300] Negotiation successful
0201 03:02:24.135005 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260201 03:02:24.141209 21450 raft_consensus.cc:1275] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Refusing update from remote peer f7cdef52faff47e6bae3b60988f108e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:24.141328 21308 raft_consensus.cc:1275] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Refusing update from remote peer f7cdef52faff47e6bae3b60988f108e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:24.141611 21629 consensus_queue.cc:1048] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:24.141749 21628 consensus_queue.cc:1048] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:24.145720 21592 mvcc.cc:204] Tried to move back new op lower bound from 7249571611199614976 to 7249571610843049984. Current Snapshot: MvccSnapshot[applied={T|T < 7249571611199614976}]
I20260201 03:02:24.168190 21005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:24.168339 21005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:24.170112 21005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b: Sending DeleteTablet for 3 replicas of tablet ea9b844f1ae747bfb5468d7569aa0d85
I20260201 03:02:24.170593 21146 tablet_service.cc:1552] Processing DeleteTablet for tablet ea9b844f1ae747bfb5468d7569aa0d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:24 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:50228
I20260201 03:02:24.170640 21288 tablet_service.cc:1552] Processing DeleteTablet for tablet ea9b844f1ae747bfb5468d7569aa0d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:24 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:33000
I20260201 03:02:24.170776 21612 tablet_replica.cc:333] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7: stopping tablet replica
I20260201 03:02:24.170811 21611 tablet_replica.cc:333] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e: stopping tablet replica
I20260201 03:02:24.170879 21612 raft_consensus.cc:2243] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:02:24.170917 21611 raft_consensus.cc:2243] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:24.171000 21611 raft_consensus.cc:2272] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:24.171041 21612 raft_consensus.cc:2272] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:24.171197 21430 tablet_service.cc:1552] Processing DeleteTablet for tablet ea9b844f1ae747bfb5468d7569aa0d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:24 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:44558
I20260201 03:02:24.171444 21613 tablet_replica.cc:333] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6: stopping tablet replica
I20260201 03:02:24.171542 21613 raft_consensus.cc:2243] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:24.171592 21611 ts_tablet_manager.cc:1916] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:24.171671 21613 raft_consensus.cc:2272] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:24.171854 21005 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:02:24.172293 21005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:02:24.172497 21613 ts_tablet_manager.cc:1916] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:24.173789 21611 ts_tablet_manager.cc:1929] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:02:24.173858 21611 log.cc:1199] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/ea9b844f1ae747bfb5468d7569aa0d85
I20260201 03:02:24.174054 21611 ts_tablet_manager.cc:1950] T ea9b844f1ae747bfb5468d7569aa0d85 P f3c41a09ad844a85982381d531c0d95e: Deleting consensus metadata
I20260201 03:02:24.174592 21613 ts_tablet_manager.cc:1929] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:02:24.174667 21613 log.cc:1199] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/ea9b844f1ae747bfb5468d7569aa0d85
I20260201 03:02:24.174757 20991 catalog_manager.cc:4985] TS f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659): tablet ea9b844f1ae747bfb5468d7569aa0d85 (table test-table [id=607810f364a446d6866cc6cd52761305]) successfully deleted
I20260201 03:02:24.174875 21613 ts_tablet_manager.cc:1950] T ea9b844f1ae747bfb5468d7569aa0d85 P 142866a5294545a9b190f6c2f66a9cc6: Deleting consensus metadata
I20260201 03:02:24.175346 20994 catalog_manager.cc:4985] TS 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125): tablet ea9b844f1ae747bfb5468d7569aa0d85 (table test-table [id=607810f364a446d6866cc6cd52761305]) successfully deleted
I20260201 03:02:24.175693 21612 ts_tablet_manager.cc:1916] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:24.177836 21612 ts_tablet_manager.cc:1929] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:02:24.177901 21612 log.cc:1199] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-0/wal/wals/ea9b844f1ae747bfb5468d7569aa0d85
I20260201 03:02:24.178125 21612 ts_tablet_manager.cc:1950] T ea9b844f1ae747bfb5468d7569aa0d85 P f7cdef52faff47e6bae3b60988f108e7: Deleting consensus metadata
I20260201 03:02:24.178292 21146 tablet_service.cc:1505] Processing CreateTablet for tablet f07e3574003c4745a04b7bd77cc364a7 (DEFAULT_TABLE table=test-table [id=dee61fc5094844f3aab626f8e9de483f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:24.178457 21288 tablet_service.cc:1505] Processing CreateTablet for tablet f07e3574003c4745a04b7bd77cc364a7 (DEFAULT_TABLE table=test-table [id=dee61fc5094844f3aab626f8e9de483f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:24.178500 21146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f07e3574003c4745a04b7bd77cc364a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:24.178663 21288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f07e3574003c4745a04b7bd77cc364a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:24.178746 20992 catalog_manager.cc:4985] TS f7cdef52faff47e6bae3b60988f108e7 (127.18.203.193:33429): tablet ea9b844f1ae747bfb5468d7569aa0d85 (table test-table [id=607810f364a446d6866cc6cd52761305]) successfully deleted
I20260201 03:02:24.180025 21624 tablet_bootstrap.cc:492] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7: Bootstrap starting.
I20260201 03:02:24.180465 21622 tablet_bootstrap.cc:492] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e: Bootstrap starting.
I20260201 03:02:24.180836 21624 tablet_bootstrap.cc:654] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:24.181095 21622 tablet_bootstrap.cc:654] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:24.181553 21430 tablet_service.cc:1505] Processing CreateTablet for tablet f07e3574003c4745a04b7bd77cc364a7 (DEFAULT_TABLE table=test-table [id=dee61fc5094844f3aab626f8e9de483f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:24.181769 21624 tablet_bootstrap.cc:492] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7: No bootstrap required, opened a new log
I20260201 03:02:24.181797 21430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f07e3574003c4745a04b7bd77cc364a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:24.181852 21624 ts_tablet_manager.cc:1403] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:24.182092 21622 tablet_bootstrap.cc:492] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e: No bootstrap required, opened a new log
I20260201 03:02:24.182176 21622 ts_tablet_manager.cc:1403] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:24.182204 21624 raft_consensus.cc:359] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:24.182313 21624 raft_consensus.cc:385] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:24.182343 21624 raft_consensus.cc:740] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cdef52faff47e6bae3b60988f108e7, State: Initialized, Role: FOLLOWER
I20260201 03:02:24.182442 21624 consensus_queue.cc:260] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:24.182564 21622 raft_consensus.cc:359] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:24.182677 21622 raft_consensus.cc:385] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:24.182679 21624 ts_tablet_manager.cc:1434] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:02:24.182705 21622 raft_consensus.cc:740] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c41a09ad844a85982381d531c0d95e, State: Initialized, Role: FOLLOWER
I20260201 03:02:24.182804 21622 consensus_queue.cc:260] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:24.183060 21622 ts_tablet_manager.cc:1434] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260201 03:02:24.183816 21623 tablet_bootstrap.cc:492] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6: Bootstrap starting.
I20260201 03:02:24.184626 21623 tablet_bootstrap.cc:654] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:24.185842 21623 tablet_bootstrap.cc:492] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6: No bootstrap required, opened a new log
I20260201 03:02:24.185913 21623 ts_tablet_manager.cc:1403] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:24.186240 21623 raft_consensus.cc:359] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:24.186326 21623 raft_consensus.cc:385] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:24.186359 21623 raft_consensus.cc:740] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142866a5294545a9b190f6c2f66a9cc6, State: Initialized, Role: FOLLOWER
I20260201 03:02:24.186471 21623 consensus_queue.cc:260] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:24.186697 21623 ts_tablet_manager.cc:1434] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260201 03:02:24.475106 21636 raft_consensus.cc:493] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:02:24.475262 21636 raft_consensus.cc:515] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:24.475740 21636 leader_election.cc:290] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659), 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125)
I20260201 03:02:24.476151 21308 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f07e3574003c4745a04b7bd77cc364a7" candidate_uuid: "f7cdef52faff47e6bae3b60988f108e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c41a09ad844a85982381d531c0d95e" is_pre_election: true
I20260201 03:02:24.476202 21450 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f07e3574003c4745a04b7bd77cc364a7" candidate_uuid: "f7cdef52faff47e6bae3b60988f108e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142866a5294545a9b190f6c2f66a9cc6" is_pre_election: true
I20260201 03:02:24.476306 21308 raft_consensus.cc:2468] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7cdef52faff47e6bae3b60988f108e7 in term 0.
I20260201 03:02:24.476339 21450 raft_consensus.cc:2468] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7cdef52faff47e6bae3b60988f108e7 in term 0.
I20260201 03:02:24.476625 21100 leader_election.cc:304] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142866a5294545a9b190f6c2f66a9cc6, f7cdef52faff47e6bae3b60988f108e7; no voters:
I20260201 03:02:24.476794 21636 raft_consensus.cc:2804] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:02:24.476866 21636 raft_consensus.cc:493] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:02:24.476904 21636 raft_consensus.cc:3060] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:24.477775 21636 raft_consensus.cc:515] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:24.478080 21636 leader_election.cc:290] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [CANDIDATE]: Term 1 election: Requested vote from peers f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659), 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125)
I20260201 03:02:24.478399 21308 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f07e3574003c4745a04b7bd77cc364a7" candidate_uuid: "f7cdef52faff47e6bae3b60988f108e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c41a09ad844a85982381d531c0d95e"
I20260201 03:02:24.478459 21450 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f07e3574003c4745a04b7bd77cc364a7" candidate_uuid: "f7cdef52faff47e6bae3b60988f108e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142866a5294545a9b190f6c2f66a9cc6"
I20260201 03:02:24.478526 21308 raft_consensus.cc:3060] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:24.478566 21450 raft_consensus.cc:3060] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:24.479579 21308 raft_consensus.cc:2468] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7cdef52faff47e6bae3b60988f108e7 in term 1.
I20260201 03:02:24.479579 21450 raft_consensus.cc:2468] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7cdef52faff47e6bae3b60988f108e7 in term 1.
I20260201 03:02:24.479904 21100 leader_election.cc:304] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142866a5294545a9b190f6c2f66a9cc6, f7cdef52faff47e6bae3b60988f108e7; no voters:
I20260201 03:02:24.480060 21636 raft_consensus.cc:2804] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:24.480149 21636 raft_consensus.cc:697] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [term 1 LEADER]: Becoming Leader. State: Replica: f7cdef52faff47e6bae3b60988f108e7, State: Running, Role: LEADER
I20260201 03:02:24.480294 21636 consensus_queue.cc:237] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } }
I20260201 03:02:24.481909 21005 catalog_manager.cc:5654] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7cdef52faff47e6bae3b60988f108e7 (127.18.203.193). New cstate: current_term: 1 leader_uuid: "f7cdef52faff47e6bae3b60988f108e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdef52faff47e6bae3b60988f108e7" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 33429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 } health_report { overall_health: UNKNOWN } } }
I20260201 03:02:24.555070 21308 raft_consensus.cc:1275] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Refusing update from remote peer f7cdef52faff47e6bae3b60988f108e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:24.555070 21450 raft_consensus.cc:1275] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Refusing update from remote peer f7cdef52faff47e6bae3b60988f108e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:24.555480 21629 consensus_queue.cc:1048] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142866a5294545a9b190f6c2f66a9cc6" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:24.555644 21636 consensus_queue.cc:1048] T f07e3574003c4745a04b7bd77cc364a7 P f7cdef52faff47e6bae3b60988f108e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c41a09ad844a85982381d531c0d95e" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260201 03:02:24.568709 21593 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260201 03:02:26.115713 21006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:26.115865 21006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:26.117982 21006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 3b7655de594a446a817e936d5252660b: Sending DeleteTablet for 3 replicas of tablet f07e3574003c4745a04b7bd77cc364a7
I20260201 03:02:26.118424 21146 tablet_service.cc:1552] Processing DeleteTablet for tablet f07e3574003c4745a04b7bd77cc364a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:26 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:50228
I20260201 03:02:26.118544 21288 tablet_service.cc:1552] Processing DeleteTablet for tablet f07e3574003c4745a04b7bd77cc364a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:26 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:33000
I20260201 03:02:26.118713 21430 tablet_service.cc:1552] Processing DeleteTablet for tablet f07e3574003c4745a04b7bd77cc364a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:26 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:44558
I20260201 03:02:26.118853 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 21081
I20260201 03:02:26.119560 21670 tablet_replica.cc:333] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6: stopping tablet replica
I20260201 03:02:26.119701 21670 raft_consensus.cc:2243] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:26.119809 21670 raft_consensus.cc:2272] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:26.120378 21670 ts_tablet_manager.cc:1916] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:26.120662 21669 tablet_replica.cc:333] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e: stopping tablet replica
I20260201 03:02:26.120774 21669 raft_consensus.cc:2243] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:26.120888 21669 raft_consensus.cc:2272] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:26.121420 21669 ts_tablet_manager.cc:1916] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:26.122705 21670 ts_tablet_manager.cc:1929] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260201 03:02:26.122773 21670 log.cc:1199] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/f07e3574003c4745a04b7bd77cc364a7
I20260201 03:02:26.123027 21670 ts_tablet_manager.cc:1950] T f07e3574003c4745a04b7bd77cc364a7 P 142866a5294545a9b190f6c2f66a9cc6: Deleting consensus metadata
I20260201 03:02:26.123574 20994 catalog_manager.cc:4985] TS 142866a5294545a9b190f6c2f66a9cc6 (127.18.203.195:41125): tablet f07e3574003c4745a04b7bd77cc364a7 (table test-table [id=dee61fc5094844f3aab626f8e9de483f]) successfully deleted
I20260201 03:02:26.123729 21669 ts_tablet_manager.cc:1929] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260201 03:02:26.123843 21669 log.cc:1199] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/f07e3574003c4745a04b7bd77cc364a7
I20260201 03:02:26.124125 21669 ts_tablet_manager.cc:1950] T f07e3574003c4745a04b7bd77cc364a7 P f3c41a09ad844a85982381d531c0d95e: Deleting consensus metadata
I20260201 03:02:26.124698 20991 catalog_manager.cc:4985] TS f3c41a09ad844a85982381d531c0d95e (127.18.203.194:41659): tablet f07e3574003c4745a04b7bd77cc364a7 (table test-table [id=dee61fc5094844f3aab626f8e9de483f]) successfully deleted
W20260201 03:02:26.128132 20992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.203.193:33429 (error 108)
I20260201 03:02:26.128182 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 21223
I20260201 03:02:26.129994 21672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:26.129434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:33429 (local address 127.0.0.1:50234)
0201 03:02:26.129846 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:26.129907 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.193:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":81,"threads_started":1}
W20260201 03:02:26.130124 20992 catalog_manager.cc:4712] TS f7cdef52faff47e6bae3b60988f108e7 (127.18.203.193:33429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f07e3574003c4745a04b7bd77cc364a7: Network error: Client connection negotiation failed: client connection to 127.18.203.193:33429: connect: Connection refused (error 111)
I20260201 03:02:26.134891 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 21365
I20260201 03:02:26.140944 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 20975
2026-02-01T03:02:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8278 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:02:26 dist-test-slave-gf59 krb5kdc[21675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:02:26 dist-test-slave-gf59 krb5kdc[21675](info): set up 2 sockets
Feb 01 03:02:26 dist-test-slave-gf59 krb5kdc[21675](info): commencing operation
krb5kdc: starting...
W20260201 03:02:27.959805 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.796s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:02:28 dist-test-slave-gf59 krb5kdc[21675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:02:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:28.095871 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:36727
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:36727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:28.201290 21691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:28.201546 21691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:28.201622 21691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:28.204931 21691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:02:28.205006 21691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:28.205039 21691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:28.205065 21691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:02:28.205096 21691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:02:28.209318 21691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:36727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:36727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:28.210400 21691 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:28.211275 21691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:28.216948 21696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:28.216948 21697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:28.217025 21691 server_base.cc:1060] running on GCE node
W20260201 03:02:28.216970 21699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:28.217679 21691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:28.218581 21691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:28.219723 21691 hybrid_clock.cc:648] HybridClock initialized: now 1769914948219714 us; error 27 us; skew 500 ppm
Feb 01 03:02:28 dist-test-slave-gf59 krb5kdc[21675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914948, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:28.222188 21691 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:02:28.223239 21691 webserver.cc:492] Webserver started at http://127.18.203.254:39927/ using document root <none> and password file <none>
I20260201 03:02:28.223793 21691 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:28.223843 21691 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:28.224004 21691 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:28.225644 21691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "dbfd8921cdca46e0bcd3a77afcbfff2b"
format_stamp: "Formatted at 2026-02-01 03:02:28 on dist-test-slave-gf59"
server_key: "0b2a7e1ee9f100e58742f6fc4ac64a5c"
server_key_iv: "823146a43fef57faefba626ec7c5c6fa"
server_key_version: "encryptionkey@0"
I20260201 03:02:28.226070 21691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "dbfd8921cdca46e0bcd3a77afcbfff2b"
format_stamp: "Formatted at 2026-02-01 03:02:28 on dist-test-slave-gf59"
server_key: "0b2a7e1ee9f100e58742f6fc4ac64a5c"
server_key_iv: "823146a43fef57faefba626ec7c5c6fa"
server_key_version: "encryptionkey@0"
I20260201 03:02:28.229334 21691 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:02:28.231370 21706 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:28.232384 21691 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:28.232492 21691 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "dbfd8921cdca46e0bcd3a77afcbfff2b"
format_stamp: "Formatted at 2026-02-01 03:02:28 on dist-test-slave-gf59"
server_key: "0b2a7e1ee9f100e58742f6fc4ac64a5c"
server_key_iv: "823146a43fef57faefba626ec7c5c6fa"
server_key_version: "encryptionkey@0"
I20260201 03:02:28.232576 21691 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:28.254729 21691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:28.257783 21691 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:28.257963 21691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:28.265029 21691 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:36727
I20260201 03:02:28.265028 21758 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:36727 every 8 connection(s)
I20260201 03:02:28.266049 21691 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:02:28.268746 21759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:28.270747 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 21691
I20260201 03:02:28.270874 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:02:28.271090 19247 external_mini_cluster.cc:1468] Setting key 21005434c3db2acfad68dcd660ec6076
I20260201 03:02:28.275066 21759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b: Bootstrap starting.
I20260201 03:02:28.277712 21759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:28.278406 21759 log.cc:826] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b: Log is configured to *not* fsync() on all Append() calls
Feb 01 03:02:28 dist-test-slave-gf59 krb5kdc[21675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:28.280490 21759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b: No bootstrap required, opened a new log
I20260201 03:02:28.283828 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:28.272316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.18.203.254:36727)
0201 03:02:28.272711 (+ 395us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:28.272722 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:28.272763 (+ 41us) server_negotiation.cc:408] Connection header received
0201 03:02:28.273391 (+ 628us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:28.273418 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:28.273785 (+ 367us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:28.274156 (+ 371us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:28.274911 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:28.276035 (+ 1124us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:28.277004 (+ 969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:28.277292 (+ 288us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:28.279287 (+ 1995us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:28.279306 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:28.279319 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:28.279359 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:28.281254 (+ 1895us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:28.281639 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:28.281645 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:28.281650 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:28.281746 (+ 96us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:28.281935 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:28.281938 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:28.281940 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:28.282376 (+ 436us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:28.282574 (+ 198us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:28.282779 (+ 205us) server_negotiation.cc:300] Negotiation successful
0201 03:02:28.283044 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
I20260201 03:02:28.283675 21759 raft_consensus.cc:359] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbfd8921cdca46e0bcd3a77afcbfff2b" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 36727 } }
I20260201 03:02:28.283938 21759 raft_consensus.cc:385] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:28.283988 21759 raft_consensus.cc:740] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbfd8921cdca46e0bcd3a77afcbfff2b, State: Initialized, Role: FOLLOWER
I20260201 03:02:28.284415 21759 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbfd8921cdca46e0bcd3a77afcbfff2b" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 36727 } }
I20260201 03:02:28.284557 21759 raft_consensus.cc:399] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:02:28.284637 21759 raft_consensus.cc:493] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:02:28.284734 21759 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:28.285831 21759 raft_consensus.cc:515] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbfd8921cdca46e0bcd3a77afcbfff2b" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 36727 } }
I20260201 03:02:28.286212 21759 leader_election.cc:304] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbfd8921cdca46e0bcd3a77afcbfff2b; no voters:
I20260201 03:02:28.286551 21759 leader_election.cc:290] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:02:28.286650 21764 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:28.286845 21764 raft_consensus.cc:697] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [term 1 LEADER]: Becoming Leader. State: Replica: dbfd8921cdca46e0bcd3a77afcbfff2b, State: Running, Role: LEADER
I20260201 03:02:28.287168 21764 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbfd8921cdca46e0bcd3a77afcbfff2b" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 36727 } }
I20260201 03:02:28.287559 21759 sys_catalog.cc:565] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:02:28.288720 21765 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbfd8921cdca46e0bcd3a77afcbfff2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbfd8921cdca46e0bcd3a77afcbfff2b" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 36727 } } }
I20260201 03:02:28.288913 21765 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:28.288744 21766 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbfd8921cdca46e0bcd3a77afcbfff2b. Latest consensus state: current_term: 1 leader_uuid: "dbfd8921cdca46e0bcd3a77afcbfff2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbfd8921cdca46e0bcd3a77afcbfff2b" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 36727 } } }
I20260201 03:02:28.289269 21766 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:28.289273 21773 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:02:28.292308 21773 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:02:28.297583 21773 catalog_manager.cc:1357] Generated new cluster ID: 1331b1c54fc24efc94a5cfff30b3e4d3
I20260201 03:02:28.297652 21773 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:02:28.313189 21773 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:02:28.314128 21773 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:02:28.325520 21773 catalog_manager.cc:6027] T 00000000000000000000000000000000 P dbfd8921cdca46e0bcd3a77afcbfff2b: Generated new TSK 0
I20260201 03:02:28.326275 21773 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:28.384723 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:36727
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:28.511823 21787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:28.512048 21787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:28.512109 21787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:28.515257 21787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:28.515318 21787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:28.515421 21787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:02:28.519526 21787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:36727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:28.520632 21787 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:28.521482 21787 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:28.527843 21792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:28.527920 21795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:28.527846 21793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:28.528081 21787 server_base.cc:1060] running on GCE node
I20260201 03:02:28.528471 21787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:28.529004 21787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:28.530149 21787 hybrid_clock.cc:648] HybridClock initialized: now 1769914948530138 us; error 29 us; skew 500 ppm
Feb 01 03:02:28 dist-test-slave-gf59 krb5kdc[21675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914948, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:28.532577 21787 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:02:28.533604 21787 webserver.cc:492] Webserver started at http://127.18.203.193:35439/ using document root <none> and password file <none>
I20260201 03:02:28.534147 21787 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:28.534200 21787 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:28.534379 21787 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:28.536123 21787 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "aa3afaea39ff4257b538d33f679ae90d"
format_stamp: "Formatted at 2026-02-01 03:02:28 on dist-test-slave-gf59"
server_key: "181522cb15e0a7b9c9d0daff00be293c"
server_key_iv: "83fa85e71e9b7eaffbe5571bc8b1c648"
server_key_version: "encryptionkey@0"
I20260201 03:02:28.536649 21787 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "aa3afaea39ff4257b538d33f679ae90d"
format_stamp: "Formatted at 2026-02-01 03:02:28 on dist-test-slave-gf59"
server_key: "181522cb15e0a7b9c9d0daff00be293c"
server_key_iv: "83fa85e71e9b7eaffbe5571bc8b1c648"
server_key_version: "encryptionkey@0"
I20260201 03:02:28.539981 21787 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:02:28.542042 21802 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:28.543046 21787 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:28.543146 21787 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "aa3afaea39ff4257b538d33f679ae90d"
format_stamp: "Formatted at 2026-02-01 03:02:28 on dist-test-slave-gf59"
server_key: "181522cb15e0a7b9c9d0daff00be293c"
server_key_iv: "83fa85e71e9b7eaffbe5571bc8b1c648"
server_key_version: "encryptionkey@0"
I20260201 03:02:28.543231 21787 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:28.559449 21787 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:28.562011 21787 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:28.562192 21787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:28.562758 21787 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:28.563657 21787 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:28.563704 21787 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:28.563755 21787 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:28.563783 21787 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:28.572433 21787 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:37429
I20260201 03:02:28.572461 21915 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:37429 every 8 connection(s)
I20260201 03:02:28.573380 21787 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
Feb 01 03:02:28 dist-test-slave-gf59 krb5kdc[21675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914948, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:28.581532 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 21787
I20260201 03:02:28.581627 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:02:28.581883 19247 external_mini_cluster.cc:1468] Setting key 323f08e13fca8d93e3faf0d52a940316
I20260201 03:02:28.585274 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:28.575047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:52119 (local address 127.18.203.254:36727)
0201 03:02:28.575222 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:28.575227 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:28.575898 (+ 671us) server_negotiation.cc:408] Connection header received
0201 03:02:28.576707 (+ 809us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:28.576710 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:28.576758 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:28.576830 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:28.578581 (+ 1751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:28.579237 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:28.579873 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:28.580006 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:28.582166 (+ 2160us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:28.582186 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:28.582188 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:28.582213 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:28.583525 (+ 1312us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:28.583976 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:28.583979 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:28.583981 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:28.584023 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:28.584284 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:28.584287 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:28.584288 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:28.584425 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:28.584504 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:28.584985 (+ 481us) server_negotiation.cc:300] Negotiation successful
0201 03:02:28.585113 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260201 03:02:28.585871 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:28.575316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:36727 (local address 127.18.203.193:52119)
0201 03:02:28.575759 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:28.575796 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:28.576530 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:28.577008 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:28.577019 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:28.577502 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:28.578437 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:28.578450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:28.579331 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:28.579334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:28.579775 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:28.579783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:28.579957 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:28.580485 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:28.580511 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:28.582018 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:28.583649 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:28.583654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:28.583666 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:28.583887 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:28.584106 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:28.584108 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:28.584110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:28.584193 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:28.584518 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:28.584525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:28.584749 (+ 224us) client_negotiation.cc:769] Sending connection context
0201 03:02:28.584925 (+ 176us) client_negotiation.cc:240] Negotiation successful
0201 03:02:28.585191 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":101,"threads_started":1}
I20260201 03:02:28.586948 21916 heartbeater.cc:344] Connected to a master server at 127.18.203.254:36727
I20260201 03:02:28.587172 21916 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:28.587692 21916 heartbeater.cc:507] Master 127.18.203.254:36727 requested a full tablet report, sending...
I20260201 03:02:28.589112 21723 ts_manager.cc:194] Registered new tserver with Master: aa3afaea39ff4257b538d33f679ae90d (127.18.203.193:37429)
I20260201 03:02:28.590325 21723 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:52119
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:28.627331 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:36727
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:28.735580 21923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:28.735862 21923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:28.735922 21923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:28.739174 21923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:28.739240 21923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:28.739318 21923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:02:28.743553 21923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:36727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:28.744628 21923 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:28.745433 21923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:28.751814 21928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:28.751884 21929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:28.751885 21931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:28.752015 21923 server_base.cc:1060] running on GCE node
I20260201 03:02:28.752331 21923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:28.752839 21923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:28.754000 21923 hybrid_clock.cc:648] HybridClock initialized: now 1769914948753990 us; error 41 us; skew 500 ppm
Feb 01 03:02:28 dist-test-slave-gf59 krb5kdc[21675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914948, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:28.756464 21923 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:02:28.757453 21923 webserver.cc:492] Webserver started at http://127.18.203.194:41467/ using document root <none> and password file <none>
I20260201 03:02:28.757946 21923 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:28.757990 21923 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:28.758136 21923 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:28.759809 21923 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "7e64b89dc32441a7a005a2ec9d59614d"
format_stamp: "Formatted at 2026-02-01 03:02:28 on dist-test-slave-gf59"
server_key: "edaf82785541fdc4d2c42731ccc0d6d2"
server_key_iv: "52dbc69beab4574df408423af5be81e9"
server_key_version: "encryptionkey@0"
I20260201 03:02:28.760246 21923 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "7e64b89dc32441a7a005a2ec9d59614d"
format_stamp: "Formatted at 2026-02-01 03:02:28 on dist-test-slave-gf59"
server_key: "edaf82785541fdc4d2c42731ccc0d6d2"
server_key_iv: "52dbc69beab4574df408423af5be81e9"
server_key_version: "encryptionkey@0"
I20260201 03:02:28.763460 21923 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260201 03:02:28.765559 21938 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:28.766551 21923 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:28.766644 21923 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "7e64b89dc32441a7a005a2ec9d59614d"
format_stamp: "Formatted at 2026-02-01 03:02:28 on dist-test-slave-gf59"
server_key: "edaf82785541fdc4d2c42731ccc0d6d2"
server_key_iv: "52dbc69beab4574df408423af5be81e9"
server_key_version: "encryptionkey@0"
I20260201 03:02:28.766717 21923 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:28.791038 21923 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:28.793848 21923 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:28.794010 21923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:28.794508 21923 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:28.795342 21923 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:28.795390 21923 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:28.795428 21923 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:28.795449 21923 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:28.803969 21923 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:44715
I20260201 03:02:28.804001 22051 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:44715 every 8 connection(s)
I20260201 03:02:28.804909 21923 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
Feb 01 03:02:28 dist-test-slave-gf59 krb5kdc[21675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914948, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:28.813930 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 21923
I20260201 03:02:28.814050 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:02:28.814342 19247 external_mini_cluster.cc:1468] Setting key c785a8527f6bd7eef8ee0d1be6eafcf8
I20260201 03:02:28.816754 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:28.806540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:55943 (local address 127.18.203.254:36727)
0201 03:02:28.806677 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:28.806681 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:28.807327 (+ 646us) server_negotiation.cc:408] Connection header received
0201 03:02:28.808181 (+ 854us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:28.808185 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:28.808243 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:28.808316 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:28.810021 (+ 1705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:28.810521 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:28.811144 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:28.811324 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:28.813372 (+ 2048us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:28.813390 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:28.813393 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:28.813420 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:28.814803 (+ 1383us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:28.815314 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:28.815317 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:28.815319 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:28.815370 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:28.815706 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:28.815709 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:28.815712 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:28.815873 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:28.815959 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:28.816473 (+ 514us) server_negotiation.cc:300] Negotiation successful
0201 03:02:28.816598 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260201 03:02:28.817348 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:28.806787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:36727 (local address 127.18.203.194:55943)
0201 03:02:28.807199 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:28.807234 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:28.807983 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:28.808478 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:28.808487 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:28.808903 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:28.809869 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:28.809883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:28.810636 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:28.810639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:28.811036 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:28.811043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:28.811256 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:28.811824 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:28.811849 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:28.813230 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:28.814919 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:28.814925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:28.814937 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:28.815222 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:28.815454 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:28.815456 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:28.815458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:28.815575 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:28.815971 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:28.815978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:28.816214 (+ 236us) client_negotiation.cc:769] Sending connection context
0201 03:02:28.816410 (+ 196us) client_negotiation.cc:240] Negotiation successful
0201 03:02:28.816666 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
I20260201 03:02:28.818395 22052 heartbeater.cc:344] Connected to a master server at 127.18.203.254:36727
I20260201 03:02:28.818643 22052 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:28.819136 22052 heartbeater.cc:507] Master 127.18.203.254:36727 requested a full tablet report, sending...
I20260201 03:02:28.820142 21723 ts_manager.cc:194] Registered new tserver with Master: 7e64b89dc32441a7a005a2ec9d59614d (127.18.203.194:44715)
I20260201 03:02:28.820876 21723 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:55943
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:28.860412 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:36727
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:28.966176 22059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:28.966408 22059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:28.966475 22059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:28.969726 22059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:28.969805 22059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:28.969911 22059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:02:28.974169 22059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:36727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:28.975274 22059 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:28.976187 22059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:28.982610 22067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:28.982647 22064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:28.982649 22065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:28.982868 22059 server_base.cc:1060] running on GCE node
I20260201 03:02:28.983289 22059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:28.983875 22059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:28.985016 22059 hybrid_clock.cc:648] HybridClock initialized: now 1769914948984997 us; error 38 us; skew 500 ppm
Feb 01 03:02:28 dist-test-slave-gf59 krb5kdc[21675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914948, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:28.987555 22059 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:02:28.988678 22059 webserver.cc:492] Webserver started at http://127.18.203.195:46363/ using document root <none> and password file <none>
I20260201 03:02:28.989235 22059 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:28.989289 22059 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:28.989500 22059 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:28.991240 22059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "d284ebeb59574397a719eafff6c6598d"
format_stamp: "Formatted at 2026-02-01 03:02:28 on dist-test-slave-gf59"
server_key: "1d9850a76c41f5b703d979cbb038d4c9"
server_key_iv: "f5f5ed9b64bab9a2d6deffa80948fe9e"
server_key_version: "encryptionkey@0"
I20260201 03:02:28.991741 22059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "d284ebeb59574397a719eafff6c6598d"
format_stamp: "Formatted at 2026-02-01 03:02:28 on dist-test-slave-gf59"
server_key: "1d9850a76c41f5b703d979cbb038d4c9"
server_key_iv: "f5f5ed9b64bab9a2d6deffa80948fe9e"
server_key_version: "encryptionkey@0"
I20260201 03:02:28.995740 22059 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260201 03:02:28.997889 22074 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:28.998870 22059 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:28.998965 22059 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "d284ebeb59574397a719eafff6c6598d"
format_stamp: "Formatted at 2026-02-01 03:02:28 on dist-test-slave-gf59"
server_key: "1d9850a76c41f5b703d979cbb038d4c9"
server_key_iv: "f5f5ed9b64bab9a2d6deffa80948fe9e"
server_key_version: "encryptionkey@0"
I20260201 03:02:28.999058 22059 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:29.015491 22059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:29.018682 22059 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:29.018879 22059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:29.019460 22059 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:29.020421 22059 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:29.020471 22059 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:29.020517 22059 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:29.020546 22059 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:29.029167 22059 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:41243
I20260201 03:02:29.029208 22187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:41243 every 8 connection(s)
I20260201 03:02:29.030158 22059 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:02:29.035956 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 22059
I20260201 03:02:29.036093 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:02:29.036381 19247 external_mini_cluster.cc:1468] Setting key 37b27a8d466bdf9d29f353e19a12fee3
Feb 01 03:02:29 dist-test-slave-gf59 krb5kdc[21675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914948, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:29.042302 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:29.031885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:39393 (local address 127.18.203.254:36727)
0201 03:02:29.032075 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:29.032080 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:29.032793 (+ 713us) server_negotiation.cc:408] Connection header received
0201 03:02:29.033641 (+ 848us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:29.033645 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:29.033699 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:29.033780 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:29.035472 (+ 1692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:29.036208 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:29.036904 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:29.037095 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:29.039476 (+ 2381us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:29.039491 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:29.039494 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:29.039522 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:29.040706 (+ 1184us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:29.041124 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:29.041127 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:29.041128 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:29.041168 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:29.041362 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:29.041364 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:29.041365 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:29.041507 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:29.041573 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:29.042063 (+ 490us) server_negotiation.cc:300] Negotiation successful
0201 03:02:29.042176 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260201 03:02:29.042951 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:29.032214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:36727 (local address 127.18.203.195:39393)
0201 03:02:29.032645 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:29.032681 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:29.033414 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:29.033938 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:29.033946 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:29.034352 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:29.035301 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:29.035316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:29.036332 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:29.036335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:29.036769 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:29.036777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:29.037014 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:29.037682 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:29.037709 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:29.039342 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:29.040800 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:29.040804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:29.040814 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:29.041047 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:29.041224 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:29.041226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:29.041228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:29.041305 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:29.041581 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:29.041588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:29.041862 (+ 274us) client_negotiation.cc:769] Sending connection context
0201 03:02:29.042040 (+ 178us) client_negotiation.cc:240] Negotiation successful
0201 03:02:29.042264 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260201 03:02:29.043952 22188 heartbeater.cc:344] Connected to a master server at 127.18.203.254:36727
I20260201 03:02:29.044205 22188 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:29.044723 22188 heartbeater.cc:507] Master 127.18.203.254:36727 requested a full tablet report, sending...
I20260201 03:02:29.045688 21723 ts_manager.cc:194] Registered new tserver with Master: d284ebeb59574397a719eafff6c6598d (127.18.203.195:41243)
I20260201 03:02:29.046240 21723 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:39393
I20260201 03:02:29.050236 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260201 03:02:29.063081 21762 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:02:29.061952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.18.203.254:36727)
0201 03:02:29.062085 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:29.062088 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:29.062106 (+ 18us) server_negotiation.cc:408] Connection header received
0201 03:02:29.062210 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:29.062212 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:29.062258 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:29.062334 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:29.062965 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56136: BlockingRecv error: recv got EOF from 127.0.0.1:56136 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260201 03:02:29.064591 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 21787
I20260201 03:02:29.070544 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 21923
I20260201 03:02:29.075793 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 22059
I20260201 03:02:29.080353 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 21691
2026-02-01T03:02:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2938 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:02:29 dist-test-slave-gf59 krb5kdc[22200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:02:29 dist-test-slave-gf59 krb5kdc[22200](info): set up 2 sockets
Feb 01 03:02:29 dist-test-slave-gf59 krb5kdc[22200](info): commencing operation
krb5kdc: starting...
W20260201 03:02:30.944013 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.842s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260201 03:02:31.377857 19247 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.338s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260201 03:02:31.498807 19247 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.121s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/test-user.keytab.
W20260201 03:02:32.132309 19247 mini_kdc.cc:290] Time spent creating keytab for test-user: real 0.633s user 0.000s sys 0.002s
Feb 01 03:02:32 dist-test-slave-gf59 krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:02:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
W20260201 03:02:33.460950 19247 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.18.203.254: real 1.306s user 0.002s sys 0.002s
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
W20260201 03:02:34.743983 19247 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.18.203.254: real 1.283s user 0.000s sys 0.004s
I20260201 03:02:34.744534 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:46345
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:34.859138 22216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:34.859396 22216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:34.859475 22216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:34.862757 22216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:02:34.862829 22216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:34.862860 22216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:34.862884 22216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:02:34.862907 22216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:02:34.867146 22216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:46345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:34.868240 22216 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:34.869097 22216 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:34.874701 22221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:34.874711 22224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:34.874706 22222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:34.874990 22216 server_base.cc:1060] running on GCE node
I20260201 03:02:34.875548 22216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:34.876468 22216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:34.877661 22216 hybrid_clock.cc:648] HybridClock initialized: now 1769914954877650 us; error 28 us; skew 500 ppm
Feb 01 03:02:34 dist-test-slave-gf59 krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914954, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:34.880358 22216 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:02:34.881472 22216 webserver.cc:492] Webserver started at http://127.18.203.254:35171/ using document root <none> and password file <none>
I20260201 03:02:34.882025 22216 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:34.882083 22216 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:34.882275 22216 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:34.884112 22216 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "ac00675c9e8748fe96a680fa3394ff24"
format_stamp: "Formatted at 2026-02-01 03:02:34 on dist-test-slave-gf59"
server_key: "1d75fb0e190a8fb496db26db014e9e58"
server_key_iv: "ce167f49a62efda9bc45e7fd2e68869c"
server_key_version: "encryptionkey@0"
I20260201 03:02:34.884613 22216 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "ac00675c9e8748fe96a680fa3394ff24"
format_stamp: "Formatted at 2026-02-01 03:02:34 on dist-test-slave-gf59"
server_key: "1d75fb0e190a8fb496db26db014e9e58"
server_key_iv: "ce167f49a62efda9bc45e7fd2e68869c"
server_key_version: "encryptionkey@0"
I20260201 03:02:34.888283 22216 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260201 03:02:34.890396 22231 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:34.891371 22216 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:02:34.891481 22216 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "ac00675c9e8748fe96a680fa3394ff24"
format_stamp: "Formatted at 2026-02-01 03:02:34 on dist-test-slave-gf59"
server_key: "1d75fb0e190a8fb496db26db014e9e58"
server_key_iv: "ce167f49a62efda9bc45e7fd2e68869c"
server_key_version: "encryptionkey@0"
I20260201 03:02:34.891566 22216 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:34.914630 22216 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:34.917390 22216 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:34.917580 22216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:34.924665 22283 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:46345 every 8 connection(s)
I20260201 03:02:34.924670 22216 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:46345
I20260201 03:02:34.925737 22216 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:02:34.928709 22284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:34.931186 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 22216
I20260201 03:02:34.931308 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:02:34.931578 19247 external_mini_cluster.cc:1468] Setting key 375fd1243320a59ebcf10cf12b64b472
I20260201 03:02:34.934937 22284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24: Bootstrap starting.
I20260201 03:02:34.937469 22284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:34.938426 22284 log.cc:826] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24: Log is configured to *not* fsync() on all Append() calls
Feb 01 03:02:34 dist-test-slave-gf59 krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:34.940496 22284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24: No bootstrap required, opened a new log
I20260201 03:02:34.943337 22284 raft_consensus.cc:359] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac00675c9e8748fe96a680fa3394ff24" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 46345 } }
I20260201 03:02:34.943544 22284 raft_consensus.cc:385] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:34.943604 22284 raft_consensus.cc:740] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac00675c9e8748fe96a680fa3394ff24, State: Initialized, Role: FOLLOWER
I20260201 03:02:34.944074 22284 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac00675c9e8748fe96a680fa3394ff24" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 46345 } }
I20260201 03:02:34.944168 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:34.932902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.18.203.254:46345)
0201 03:02:34.933291 (+ 389us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:34.933301 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:34.933337 (+ 36us) server_negotiation.cc:408] Connection header received
0201 03:02:34.933907 (+ 570us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:34.933925 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:34.934246 (+ 321us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:34.934532 (+ 286us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:34.935642 (+ 1110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:34.936420 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:34.937270 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:34.937481 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:34.939943 (+ 2462us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:34.939963 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:34.939974 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:34.940009 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:34.941658 (+ 1649us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:34.942110 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:34.942115 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:34.942119 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:34.942195 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:34.942412 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:34.942414 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:34.942415 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:34.942765 (+ 350us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:34.942934 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:34.943196 (+ 262us) server_negotiation.cc:300] Negotiation successful
0201 03:02:34.943446 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260201 03:02:34.944209 22284 raft_consensus.cc:399] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:02:34.944271 22284 raft_consensus.cc:493] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:02:34.944386 22284 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:34.945422 22284 raft_consensus.cc:515] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac00675c9e8748fe96a680fa3394ff24" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 46345 } }
I20260201 03:02:34.945756 22284 leader_election.cc:304] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac00675c9e8748fe96a680fa3394ff24; no voters:
I20260201 03:02:34.946177 22284 leader_election.cc:290] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:02:34.946300 22289 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:34.946601 22289 raft_consensus.cc:697] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [term 1 LEADER]: Becoming Leader. State: Replica: ac00675c9e8748fe96a680fa3394ff24, State: Running, Role: LEADER
I20260201 03:02:34.947029 22289 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac00675c9e8748fe96a680fa3394ff24" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 46345 } }
I20260201 03:02:34.947360 22284 sys_catalog.cc:565] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:02:34.948578 22291 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac00675c9e8748fe96a680fa3394ff24. Latest consensus state: current_term: 1 leader_uuid: "ac00675c9e8748fe96a680fa3394ff24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac00675c9e8748fe96a680fa3394ff24" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 46345 } } }
I20260201 03:02:34.948697 22291 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:34.949056 22298 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:02:34.950021 22290 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac00675c9e8748fe96a680fa3394ff24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac00675c9e8748fe96a680fa3394ff24" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 46345 } } }
I20260201 03:02:34.950099 22290 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:34.952219 22298 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:02:34.957233 22298 catalog_manager.cc:1357] Generated new cluster ID: 0c16b54005594f58bc047062ca6cdf3a
I20260201 03:02:34.957310 22298 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:02:34.967152 22298 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:02:34.968174 22298 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:02:34.973443 22298 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ac00675c9e8748fe96a680fa3394ff24: Generated new TSK 0
I20260201 03:02:34.974349 22298 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:35.031926 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:46345
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:35.139128 22312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:35.139396 22312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:35.139480 22312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:35.142808 22312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:35.142896 22312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:35.143002 22312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:02:35.147207 22312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:35.148324 22312 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:35.149240 22312 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:35.155912 22317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:35.156054 22312 server_base.cc:1060] running on GCE node
W20260201 03:02:35.155920 22320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:35.155920 22318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:35.156543 22312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:35.157178 22312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:35.158339 22312 hybrid_clock.cc:648] HybridClock initialized: now 1769914955158322 us; error 39 us; skew 500 ppm
Feb 01 03:02:35 dist-test-slave-gf59 krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914955, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:35.160945 22312 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:02:35.161962 22312 webserver.cc:492] Webserver started at http://127.18.203.193:40481/ using document root <none> and password file <none>
I20260201 03:02:35.162478 22312 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:35.162524 22312 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:35.162709 22312 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:35.164438 22312 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "23e84afbedc14f37ba4c2331c190f66b"
format_stamp: "Formatted at 2026-02-01 03:02:35 on dist-test-slave-gf59"
server_key: "5a6ffb22be7909ae35c20510daea3dcb"
server_key_iv: "0184c2b067cc5d3e0c47fad563733130"
server_key_version: "encryptionkey@0"
I20260201 03:02:35.164881 22312 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "23e84afbedc14f37ba4c2331c190f66b"
format_stamp: "Formatted at 2026-02-01 03:02:35 on dist-test-slave-gf59"
server_key: "5a6ffb22be7909ae35c20510daea3dcb"
server_key_iv: "0184c2b067cc5d3e0c47fad563733130"
server_key_version: "encryptionkey@0"
I20260201 03:02:35.168452 22312 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:02:35.170748 22327 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:35.171867 22312 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:02:35.171977 22312 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "23e84afbedc14f37ba4c2331c190f66b"
format_stamp: "Formatted at 2026-02-01 03:02:35 on dist-test-slave-gf59"
server_key: "5a6ffb22be7909ae35c20510daea3dcb"
server_key_iv: "0184c2b067cc5d3e0c47fad563733130"
server_key_version: "encryptionkey@0"
I20260201 03:02:35.172091 22312 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:35.192373 22312 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:35.195189 22312 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:35.195374 22312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:35.195966 22312 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:35.196866 22312 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:35.196915 22312 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:35.196957 22312 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:35.196985 22312 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:35.205629 22312 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:41915
I20260201 03:02:35.205662 22440 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:41915 every 8 connection(s)
I20260201 03:02:35.206581 22312 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:02:35.208392 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 22312
I20260201 03:02:35.208508 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:02:35.208760 19247 external_mini_cluster.cc:1468] Setting key 7045d108945323841fe82f3af0c017e1
Feb 01 03:02:35 dist-test-slave-gf59 krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914955, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
I20260201 03:02:35.219774 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:35.208397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:37727 (local address 127.18.203.254:46345)
0201 03:02:35.208568 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:35.208573 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:35.209225 (+ 652us) server_negotiation.cc:408] Connection header received
0201 03:02:35.210075 (+ 850us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:35.210078 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:35.210132 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:35.210203 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:35.212378 (+ 2175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:35.213158 (+ 780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:35.213775 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:35.213965 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:35.216452 (+ 2487us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:35.216471 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:35.216474 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:35.216503 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:35.217863 (+ 1360us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:35.218416 (+ 553us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:35.218419 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:35.218420 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:35.218457 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:35.218758 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:35.218760 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:35.218761 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:35.218922 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:35.218993 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:35.219457 (+ 464us) server_negotiation.cc:300] Negotiation successful
0201 03:02:35.219579 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260201 03:02:35.220391 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:35.208644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:46345 (local address 127.18.203.193:37727)
0201 03:02:35.209073 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:35.209115 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:35.209891 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:35.210395 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:35.210406 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:35.210937 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:35.212199 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:35.212219 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:35.213276 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:35.213279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:35.213664 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:35.213671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:35.213839 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:35.214401 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:35.214427 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:35.216267 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:35.218003 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:35.218009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:35.218022 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:35.218324 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:35.218544 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:35.218548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:35.218550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:35.218672 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:35.219011 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:35.219017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:35.219251 (+ 234us) client_negotiation.cc:769] Sending connection context
0201 03:02:35.219430 (+ 179us) client_negotiation.cc:240] Negotiation successful
0201 03:02:35.219692 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
I20260201 03:02:35.221617 22441 heartbeater.cc:344] Connected to a master server at 127.18.203.254:46345
I20260201 03:02:35.221873 22441 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:35.222388 22441 heartbeater.cc:507] Master 127.18.203.254:46345 requested a full tablet report, sending...
I20260201 03:02:35.223913 22248 ts_manager.cc:194] Registered new tserver with Master: 23e84afbedc14f37ba4c2331c190f66b (127.18.203.193:41915)
I20260201 03:02:35.225441 22248 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:37727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:35.259035 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:46345
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:35.364961 22448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:35.365211 22448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:35.365296 22448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:35.368547 22448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:35.368618 22448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:35.368722 22448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:02:35.372825 22448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:35.373847 22448 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:35.374675 22448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:35.380841 22456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:35.380904 22454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:35.380916 22453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:35.381103 22448 server_base.cc:1060] running on GCE node
I20260201 03:02:35.381459 22448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:35.381955 22448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:35.383097 22448 hybrid_clock.cc:648] HybridClock initialized: now 1769914955383079 us; error 35 us; skew 500 ppm
Feb 01 03:02:35 dist-test-slave-gf59 krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914955, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:35.385640 22448 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:02:35.386687 22448 webserver.cc:492] Webserver started at http://127.18.203.194:40105/ using document root <none> and password file <none>
I20260201 03:02:35.387223 22448 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:35.387274 22448 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:35.387454 22448 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:35.389183 22448 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "edd4dec100cf46a6b883cc22db6b949e"
format_stamp: "Formatted at 2026-02-01 03:02:35 on dist-test-slave-gf59"
server_key: "fde1c9a0f49040715fda5be7fe446a0b"
server_key_iv: "bb329c8f19716c68fb369d1b5290cae6"
server_key_version: "encryptionkey@0"
I20260201 03:02:35.389645 22448 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "edd4dec100cf46a6b883cc22db6b949e"
format_stamp: "Formatted at 2026-02-01 03:02:35 on dist-test-slave-gf59"
server_key: "fde1c9a0f49040715fda5be7fe446a0b"
server_key_iv: "bb329c8f19716c68fb369d1b5290cae6"
server_key_version: "encryptionkey@0"
I20260201 03:02:35.392949 22448 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:02:35.395089 22463 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:35.396162 22448 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:02:35.396263 22448 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "edd4dec100cf46a6b883cc22db6b949e"
format_stamp: "Formatted at 2026-02-01 03:02:35 on dist-test-slave-gf59"
server_key: "fde1c9a0f49040715fda5be7fe446a0b"
server_key_iv: "bb329c8f19716c68fb369d1b5290cae6"
server_key_version: "encryptionkey@0"
I20260201 03:02:35.396347 22448 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:35.427327 22448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:35.430225 22448 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:35.430416 22448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:35.430965 22448 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:35.431912 22448 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:35.431964 22448 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:35.432019 22448 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:35.432045 22448 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:35.441057 22448 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:34425
I20260201 03:02:35.441082 22576 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:34425 every 8 connection(s)
I20260201 03:02:35.442020 22448 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:02:35.445075 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 22448
I20260201 03:02:35.445179 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:02:35.445393 19247 external_mini_cluster.cc:1468] Setting key d7cbe38adeba6a5b75f071cdd46e4021
Feb 01 03:02:35 dist-test-slave-gf59 krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914955, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
I20260201 03:02:35.454893 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:35.443841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:47003 (local address 127.18.203.254:46345)
0201 03:02:35.443990 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:35.443995 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:35.444677 (+ 682us) server_negotiation.cc:408] Connection header received
0201 03:02:35.445536 (+ 859us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:35.445540 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:35.445600 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:35.445693 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:35.447478 (+ 1785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:35.448234 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:35.448881 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:35.449057 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:35.451663 (+ 2606us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:35.451680 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:35.451683 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:35.451709 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:35.453062 (+ 1353us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:35.453524 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:35.453527 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:35.453529 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:35.453572 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:35.453834 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:35.453837 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:35.453839 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:35.453985 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:35.454071 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:35.454577 (+ 506us) server_negotiation.cc:300] Negotiation successful
0201 03:02:35.454718 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260201 03:02:35.455520 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:35.444111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:46345 (local address 127.18.203.194:47003)
0201 03:02:35.444531 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:35.444571 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:35.445345 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:35.445887 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:35.445898 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:35.446308 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:35.447329 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:35.447344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:35.448365 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:35.448369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:35.448767 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:35.448775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:35.449001 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:35.449640 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:35.449665 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:35.451486 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:35.453177 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:35.453183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:35.453196 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:35.453426 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:35.453652 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:35.453655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:35.453657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:35.453744 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:35.454092 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:35.454099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:35.454336 (+ 237us) client_negotiation.cc:769] Sending connection context
0201 03:02:35.454526 (+ 190us) client_negotiation.cc:240] Negotiation successful
0201 03:02:35.454798 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20260201 03:02:35.456636 22577 heartbeater.cc:344] Connected to a master server at 127.18.203.254:46345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
I20260201 03:02:35.456918 22577 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:35.457532 22577 heartbeater.cc:507] Master 127.18.203.254:46345 requested a full tablet report, sending...
I20260201 03:02:35.458582 22248 ts_manager.cc:194] Registered new tserver with Master: edd4dec100cf46a6b883cc22db6b949e (127.18.203.194:34425)
I20260201 03:02:35.459168 22248 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:47003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:35.491680 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:46345
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:35.598168 22584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:35.598433 22584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:35.598495 22584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:35.601840 22584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:35.601926 22584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:35.602033 22584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:02:35.606343 22584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:35.607461 22584 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:35.608385 22584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:35.614814 22592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:35.614794 22589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:35.614858 22590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:35.615199 22584 server_base.cc:1060] running on GCE node
I20260201 03:02:35.615550 22584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:35.616151 22584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:35.617296 22584 hybrid_clock.cc:648] HybridClock initialized: now 1769914955617281 us; error 31 us; skew 500 ppm
Feb 01 03:02:35 dist-test-slave-gf59 krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914955, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:35.620005 22584 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:02:35.621057 22584 webserver.cc:492] Webserver started at http://127.18.203.195:39535/ using document root <none> and password file <none>
I20260201 03:02:35.621569 22584 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:35.621629 22584 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:35.621848 22584 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:35.623577 22584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "86677ccccc814d35aaa3f84d88730f23"
format_stamp: "Formatted at 2026-02-01 03:02:35 on dist-test-slave-gf59"
server_key: "951347dae11d9a9d634e96eecb3a4326"
server_key_iv: "cb391227dadd8f3432c8eaa1055a5391"
server_key_version: "encryptionkey@0"
I20260201 03:02:35.624070 22584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "86677ccccc814d35aaa3f84d88730f23"
format_stamp: "Formatted at 2026-02-01 03:02:35 on dist-test-slave-gf59"
server_key: "951347dae11d9a9d634e96eecb3a4326"
server_key_iv: "cb391227dadd8f3432c8eaa1055a5391"
server_key_version: "encryptionkey@0"
I20260201 03:02:35.627274 22584 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260201 03:02:35.629431 22599 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:35.630406 22584 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:35.630501 22584 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "86677ccccc814d35aaa3f84d88730f23"
format_stamp: "Formatted at 2026-02-01 03:02:35 on dist-test-slave-gf59"
server_key: "951347dae11d9a9d634e96eecb3a4326"
server_key_iv: "cb391227dadd8f3432c8eaa1055a5391"
server_key_version: "encryptionkey@0"
I20260201 03:02:35.630576 22584 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:35.646970 22584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:35.649857 22584 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:35.650053 22584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:35.650578 22584 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:35.651453 22584 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:35.651501 22584 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:35.651539 22584 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:35.651559 22584 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:35.660120 22584 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:43737
I20260201 03:02:35.660153 22712 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:43737 every 8 connection(s)
I20260201 03:02:35.661105 22584 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:02:35.667423 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 22584
I20260201 03:02:35.667553 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:02:35.667845 19247 external_mini_cluster.cc:1468] Setting key bf396df0cb37b0b74964bcc4e110690c
Feb 01 03:02:35 dist-test-slave-gf59 krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914955, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:35.673187 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:35.662858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:47777 (local address 127.18.203.254:46345)
0201 03:02:35.663097 (+ 239us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:35.663101 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:35.663687 (+ 586us) server_negotiation.cc:408] Connection header received
0201 03:02:35.664511 (+ 824us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:35.664514 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:35.664564 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:35.664643 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:35.666291 (+ 1648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:35.666790 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:35.667406 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:35.667596 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:35.670018 (+ 2422us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:35.670035 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:35.670038 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:35.670071 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:35.671432 (+ 1361us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:35.671916 (+ 484us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:35.671920 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:35.671922 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:35.671973 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:35.672197 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:35.672199 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:35.672200 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:35.672349 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:35.672413 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:35.672936 (+ 523us) server_negotiation.cc:300] Negotiation successful
0201 03:02:35.673055 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260201 03:02:35.673753 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:35.663103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:46345 (local address 127.18.203.195:47777)
0201 03:02:35.663524 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:35.663560 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:35.664332 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:35.664798 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:35.664806 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:35.665204 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:35.666163 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:35.666177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:35.666895 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:35.666898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:35.667295 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:35.667302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:35.667471 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:35.668077 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:35.668102 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:35.669859 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:35.671536 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:35.671541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:35.671551 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:35.671829 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:35.672047 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:35.672049 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:35.672050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:35.672132 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:35.672415 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:35.672420 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:35.672652 (+ 232us) client_negotiation.cc:769] Sending connection context
0201 03:02:35.672821 (+ 169us) client_negotiation.cc:240] Negotiation successful
0201 03:02:35.673041 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":90,"threads_started":1}
I20260201 03:02:35.674718 22713 heartbeater.cc:344] Connected to a master server at 127.18.203.254:46345
I20260201 03:02:35.674963 22713 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:35.675484 22713 heartbeater.cc:507] Master 127.18.203.254:46345 requested a full tablet report, sending...
I20260201 03:02:35.676534 22248 ts_manager.cc:194] Registered new tserver with Master: 86677ccccc814d35aaa3f84d88730f23 (127.18.203.195:43737)
I20260201 03:02:35.677089 22248 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:47777
I20260201 03:02:35.681720 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260201 03:02:35.814632 22287 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:02:35.810926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.18.203.254:46345)
0201 03:02:35.811059 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:35.811063 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:35.812074 (+ 1011us) server_negotiation.cc:408] Connection header received
0201 03:02:35.813004 (+ 930us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:35.813008 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:35.813072 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:35.813166 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:35.814476 (+ 1310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42684: BlockingRecv error: recv got EOF from 127.0.0.1:42684 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260201 03:02:35.815984 22287 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:02:35.815175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.18.203.254:46345)
0201 03:02:35.815312 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:35.815316 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:35.815329 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:02:35.815394 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:35.815397 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:35.815439 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:35.815505 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:35.815900 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42700: BlockingRecv error: recv got EOF from 127.0.0.1:42700 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260201 03:02:35.816951 22287 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:02:35.816237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.18.203.254:46345)
0201 03:02:35.816332 (+ 95us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:35.816336 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:35.816349 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:02:35.816443 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:35.816446 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:35.816484 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:35.816544 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:35.816868 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42716: BlockingRecv error: recv got EOF from 127.0.0.1:42716 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260201 03:02:35.820600 22287 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:02:35.819994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.18.203.254:46345)
0201 03:02:35.820093 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:35.820097 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:35.820108 (+ 11us) server_negotiation.cc:408] Connection header received
0201 03:02:35.820147 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:35.820149 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:35.820184 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:35.820248 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:35.820531 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42726: BlockingRecv error: recv got EOF from 127.0.0.1:42726 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260201 03:02:35.828935 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 22312
I20260201 03:02:35.834879 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 22448
I20260201 03:02:35.839974 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 22584
I20260201 03:02:35.845167 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 22216
2026-02-01T03:02:35Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (6765 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:02:35 dist-test-slave-gf59 krb5kdc[22736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:02:35 dist-test-slave-gf59 krb5kdc[22736](info): set up 2 sockets
Feb 01 03:02:35 dist-test-slave-gf59 krb5kdc[22736](info): commencing operation
krb5kdc: starting...
W20260201 03:02:37.679867 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.812s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:02:37 dist-test-slave-gf59 krb5kdc[22736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:02:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:02:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:37.820890 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:45061
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:45061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:37.930967 22752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:37.931246 22752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:37.931326 22752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:37.934710 22752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:02:37.934818 22752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:37.934851 22752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:37.934875 22752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:02:37.934899 22752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:02:37.939507 22752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:45061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:45061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:37.940750 22752 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:37.941655 22752 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:37.947599 22757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:37.947710 22752 server_base.cc:1060] running on GCE node
W20260201 03:02:37.947599 22760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:37.947599 22758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:37.948318 22752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:37.949271 22752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:37.950428 22752 hybrid_clock.cc:648] HybridClock initialized: now 1769914957950414 us; error 38 us; skew 500 ppm
Feb 01 03:02:37 dist-test-slave-gf59 krb5kdc[22736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914957, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:37.953164 22752 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:02:37.954342 22752 webserver.cc:492] Webserver started at http://127.18.203.254:37329/ using document root <none> and password file <none>
I20260201 03:02:37.954916 22752 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:37.954970 22752 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:37.955163 22752 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:37.956992 22752 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "81683bda2fae4f30ad38ff2169a2a431"
format_stamp: "Formatted at 2026-02-01 03:02:37 on dist-test-slave-gf59"
server_key: "ac39bd88180f47bcc1f47a4498bdf7ba"
server_key_iv: "a6392a2ec6ea140ed683d02b0a93d2ee"
server_key_version: "encryptionkey@0"
I20260201 03:02:37.957499 22752 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "81683bda2fae4f30ad38ff2169a2a431"
format_stamp: "Formatted at 2026-02-01 03:02:37 on dist-test-slave-gf59"
server_key: "ac39bd88180f47bcc1f47a4498bdf7ba"
server_key_iv: "a6392a2ec6ea140ed683d02b0a93d2ee"
server_key_version: "encryptionkey@0"
I20260201 03:02:37.960999 22752 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:02:37.963274 22767 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:37.964426 22752 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:37.964551 22752 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "81683bda2fae4f30ad38ff2169a2a431"
format_stamp: "Formatted at 2026-02-01 03:02:37 on dist-test-slave-gf59"
server_key: "ac39bd88180f47bcc1f47a4498bdf7ba"
server_key_iv: "a6392a2ec6ea140ed683d02b0a93d2ee"
server_key_version: "encryptionkey@0"
I20260201 03:02:37.964651 22752 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:37.988426 22752 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:37.995186 22752 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:37.995384 22752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:38.002907 22819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:45061 every 8 connection(s)
I20260201 03:02:38.002923 22752 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:45061
I20260201 03:02:38.004081 22752 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:02:38.006897 22820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:38.007038 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 22752
I20260201 03:02:38.007156 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:02:38.007424 19247 external_mini_cluster.cc:1468] Setting key 861397a232256d96ebde506eb297dd90
I20260201 03:02:38.013788 22820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431: Bootstrap starting.
Feb 01 03:02:38 dist-test-slave-gf59 krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:38.016369 22820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:38.017277 22820 log.cc:826] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:38.019381 22820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431: No bootstrap required, opened a new log
I20260201 03:02:38.020599 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:38.008632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.18.203.254:45061)
0201 03:02:38.009029 (+ 397us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:38.009039 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:38.009076 (+ 37us) server_negotiation.cc:408] Connection header received
0201 03:02:38.009947 (+ 871us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:38.009970 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:38.010293 (+ 323us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:38.010589 (+ 296us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:38.011666 (+ 1077us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:38.012475 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:38.013334 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:38.013559 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:38.016020 (+ 2461us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:38.016038 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:38.016049 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:38.016087 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:38.017996 (+ 1909us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:38.018405 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:38.018411 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:38.018416 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:38.018511 (+ 95us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:38.018705 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:38.018709 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:38.018711 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:38.019151 (+ 440us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:38.019343 (+ 192us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:38.019541 (+ 198us) server_negotiation.cc:300] Negotiation successful
0201 03:02:38.019839 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260201 03:02:38.022486 22820 raft_consensus.cc:359] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81683bda2fae4f30ad38ff2169a2a431" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 45061 } }
I20260201 03:02:38.022702 22820 raft_consensus.cc:385] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:38.022763 22820 raft_consensus.cc:740] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81683bda2fae4f30ad38ff2169a2a431, State: Initialized, Role: FOLLOWER
I20260201 03:02:38.023198 22820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81683bda2fae4f30ad38ff2169a2a431" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 45061 } }
I20260201 03:02:38.023351 22820 raft_consensus.cc:399] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:02:38.023346 19247 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260201 03:02:38.023417 22820 raft_consensus.cc:493] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:02:38.023514 22820 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:38.024715 22820 raft_consensus.cc:515] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81683bda2fae4f30ad38ff2169a2a431" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 45061 } }
I20260201 03:02:38.025168 22820 leader_election.cc:304] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81683bda2fae4f30ad38ff2169a2a431; no voters:
I20260201 03:02:38.025560 22820 leader_election.cc:290] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:02:38.026491 22827 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:38.026762 22827 raft_consensus.cc:697] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [term 1 LEADER]: Becoming Leader. State: Replica: 81683bda2fae4f30ad38ff2169a2a431, State: Running, Role: LEADER
I20260201 03:02:38.027120 22827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81683bda2fae4f30ad38ff2169a2a431" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 45061 } }
I20260201 03:02:38.027242 22820 sys_catalog.cc:565] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:02:38.028507 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81683bda2fae4f30ad38ff2169a2a431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81683bda2fae4f30ad38ff2169a2a431" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 45061 } } }
I20260201 03:02:38.028613 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:38.028807 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81683bda2fae4f30ad38ff2169a2a431. Latest consensus state: current_term: 1 leader_uuid: "81683bda2fae4f30ad38ff2169a2a431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81683bda2fae4f30ad38ff2169a2a431" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 45061 } } }
I20260201 03:02:38.028872 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:38.029222 22835 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:02:38.032419 22835 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:02:38.038193 22835 catalog_manager.cc:1357] Generated new cluster ID: 9e02a3a1e5bd4298ab7387240c601d28
I20260201 03:02:38.038278 22835 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:02:38.045154 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:38.041262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.18.203.254:45061)
0201 03:02:38.041382 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:38.041386 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:38.041401 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:02:38.041445 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:38.041448 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:38.041493 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:38.041562 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:38.042262 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:38.042759 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:38.043429 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:38.043656 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:38.044182 (+ 526us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:38.044201 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:38.044204 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:38.044669 (+ 465us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:38.044701 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:38.044718 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:38.044802 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:38.044934 (+ 132us) server_negotiation.cc:300] Negotiation successful
0201 03:02:38.044995 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260201 03:02:38.048231 22835 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:02:38.049176 22835 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:02:38.056581 22835 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 81683bda2fae4f30ad38ff2169a2a431: Generated new TSK 0
I20260201 03:02:38.057348 22835 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:02:38.068475 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 22752
2026-02-01T03:02:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2222 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:02:38 dist-test-slave-gf59 krb5kdc[22853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:02:38 dist-test-slave-gf59 krb5kdc[22853](info): set up 2 sockets
Feb 01 03:02:38 dist-test-slave-gf59 krb5kdc[22853](info): commencing operation
krb5kdc: starting...
W20260201 03:02:39.904023 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.814s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:02:39 dist-test-slave-gf59 krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:02:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:02:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:40.045975 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:33943
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:33943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:40.156351 22869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:40.156635 22869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:40.156718 22869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:40.160109 22869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:02:40.160194 22869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:40.160228 22869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:40.160251 22869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:02:40.160275 22869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:02:40.164815 22869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:33943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:33943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:40.165982 22869 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:40.166862 22869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:40.172668 22877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:40.172757 22875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:40.172662 22874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:40.172922 22869 server_base.cc:1060] running on GCE node
I20260201 03:02:40.173473 22869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:40.174376 22869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:40.175519 22869 hybrid_clock.cc:648] HybridClock initialized: now 1769914960175508 us; error 30 us; skew 500 ppm
Feb 01 03:02:40 dist-test-slave-gf59 krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914960, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:40.178225 22869 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:02:40.179353 22869 webserver.cc:492] Webserver started at http://127.18.203.254:40745/ using document root <none> and password file <none>
I20260201 03:02:40.179955 22869 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:40.180012 22869 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:40.180194 22869 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:40.181881 22869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "278a0b7a1ed348708533140b4cdef0a0"
format_stamp: "Formatted at 2026-02-01 03:02:40 on dist-test-slave-gf59"
server_key: "323c4671370fe3a07ce5b02e5168c694"
server_key_iv: "7759051154ad49e40e6b45e7796f4db2"
server_key_version: "encryptionkey@0"
I20260201 03:02:40.182374 22869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "278a0b7a1ed348708533140b4cdef0a0"
format_stamp: "Formatted at 2026-02-01 03:02:40 on dist-test-slave-gf59"
server_key: "323c4671370fe3a07ce5b02e5168c694"
server_key_iv: "7759051154ad49e40e6b45e7796f4db2"
server_key_version: "encryptionkey@0"
I20260201 03:02:40.185815 22869 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260201 03:02:40.187983 22884 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:40.188992 22869 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:02:40.189103 22869 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "278a0b7a1ed348708533140b4cdef0a0"
format_stamp: "Formatted at 2026-02-01 03:02:40 on dist-test-slave-gf59"
server_key: "323c4671370fe3a07ce5b02e5168c694"
server_key_iv: "7759051154ad49e40e6b45e7796f4db2"
server_key_version: "encryptionkey@0"
I20260201 03:02:40.189194 22869 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:40.206310 22869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:40.209005 22869 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:40.209198 22869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:40.216336 22936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:33943 every 8 connection(s)
I20260201 03:02:40.216347 22869 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:33943
I20260201 03:02:40.217417 22869 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:02:40.220242 22937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:40.221947 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 22869
I20260201 03:02:40.222085 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:02:40.222324 19247 external_mini_cluster.cc:1468] Setting key 18166c5b1d25c98a56cf9a047b42ecbe
I20260201 03:02:40.226715 22937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0: Bootstrap starting.
I20260201 03:02:40.229192 22937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:40.230013 22937 log.cc:826] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0: Log is configured to *not* fsync() on all Append() calls
Feb 01 03:02:40 dist-test-slave-gf59 krb5kdc[22853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:40.232288 22937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0: No bootstrap required, opened a new log
I20260201 03:02:40.235503 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:40.223678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.18.203.254:33943)
0201 03:02:40.224113 (+ 435us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:40.224125 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:40.224169 (+ 44us) server_negotiation.cc:408] Connection header received
0201 03:02:40.224832 (+ 663us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:40.224858 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:40.225249 (+ 391us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:40.225623 (+ 374us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:40.226430 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:40.227570 (+ 1140us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:40.228249 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:40.228522 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:40.230815 (+ 2293us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:40.230832 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:40.230845 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:40.230897 (+ 52us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:40.232860 (+ 1963us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:40.233278 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:40.233283 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:40.233289 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:40.233388 (+ 99us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:40.233581 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:40.233584 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:40.233586 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:40.234051 (+ 465us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:40.234264 (+ 213us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:40.234433 (+ 169us) server_negotiation.cc:300] Negotiation successful
0201 03:02:40.234703 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":122,"threads_started":1}
I20260201 03:02:40.235503 22937 raft_consensus.cc:359] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278a0b7a1ed348708533140b4cdef0a0" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 33943 } }
I20260201 03:02:40.235860 22937 raft_consensus.cc:385] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:40.235917 22937 raft_consensus.cc:740] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278a0b7a1ed348708533140b4cdef0a0, State: Initialized, Role: FOLLOWER
I20260201 03:02:40.236361 22937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278a0b7a1ed348708533140b4cdef0a0" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 33943 } }
I20260201 03:02:40.236480 22937 raft_consensus.cc:399] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:02:40.236541 22937 raft_consensus.cc:493] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:02:40.236625 22937 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:40.237464 22937 raft_consensus.cc:515] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278a0b7a1ed348708533140b4cdef0a0" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 33943 } }
I20260201 03:02:40.237813 22937 leader_election.cc:304] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 278a0b7a1ed348708533140b4cdef0a0; no voters:
I20260201 03:02:40.238099 22937 leader_election.cc:290] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:02:40.238202 22942 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:40.238394 22942 raft_consensus.cc:697] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 278a0b7a1ed348708533140b4cdef0a0, State: Running, Role: LEADER
I20260201 03:02:40.238720 22942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278a0b7a1ed348708533140b4cdef0a0" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 33943 } }
I20260201 03:02:40.239231 22937 sys_catalog.cc:565] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:02:40.240693 22944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 278a0b7a1ed348708533140b4cdef0a0. Latest consensus state: current_term: 1 leader_uuid: "278a0b7a1ed348708533140b4cdef0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278a0b7a1ed348708533140b4cdef0a0" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 33943 } } }
I20260201 03:02:40.240818 22944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:40.241122 22943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "278a0b7a1ed348708533140b4cdef0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278a0b7a1ed348708533140b4cdef0a0" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 33943 } } }
I20260201 03:02:40.241204 22943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:40.241214 22951 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:02:40.244315 22951 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:02:40.249819 22951 catalog_manager.cc:1357] Generated new cluster ID: 36e1ede76ff247fcbdb042789a24c9aa
I20260201 03:02:40.249964 22951 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:02:40.265201 22951 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:02:40.266180 22951 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:02:40.271597 22951 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 278a0b7a1ed348708533140b4cdef0a0: Generated new TSK 0
I20260201 03:02:40.272330 22951 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:40.334761 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:33943
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:40.474299 22965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:40.474572 22965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:40.474640 22965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:40.478022 22965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:40.478106 22965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:40.478217 22965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:02:40.482625 22965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:33943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:40.483820 22965 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:40.484724 22965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:40.491266 22970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:40.491313 22973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:40.491313 22971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:40.491564 22965 server_base.cc:1060] running on GCE node
I20260201 03:02:40.491936 22965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:40.492492 22965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:40.493634 22965 hybrid_clock.cc:648] HybridClock initialized: now 1769914960493615 us; error 36 us; skew 500 ppm
Feb 01 03:02:40 dist-test-slave-gf59 krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914960, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:40.496253 22965 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:02:40.497363 22965 webserver.cc:492] Webserver started at http://127.18.203.193:40845/ using document root <none> and password file <none>
I20260201 03:02:40.497928 22965 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:40.497985 22965 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:40.498179 22965 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:40.499970 22965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "e643f799b1e84cae861f7af7a1344a34"
format_stamp: "Formatted at 2026-02-01 03:02:40 on dist-test-slave-gf59"
server_key: "9d7e7e870200b6b1dbac72614aaf3879"
server_key_iv: "c2a6422ceddc9029c19b5bd13cc5c377"
server_key_version: "encryptionkey@0"
I20260201 03:02:40.500469 22965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "e643f799b1e84cae861f7af7a1344a34"
format_stamp: "Formatted at 2026-02-01 03:02:40 on dist-test-slave-gf59"
server_key: "9d7e7e870200b6b1dbac72614aaf3879"
server_key_iv: "c2a6422ceddc9029c19b5bd13cc5c377"
server_key_version: "encryptionkey@0"
I20260201 03:02:40.503858 22965 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260201 03:02:40.506054 22980 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:40.507086 22965 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:40.507207 22965 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "e643f799b1e84cae861f7af7a1344a34"
format_stamp: "Formatted at 2026-02-01 03:02:40 on dist-test-slave-gf59"
server_key: "9d7e7e870200b6b1dbac72614aaf3879"
server_key_iv: "c2a6422ceddc9029c19b5bd13cc5c377"
server_key_version: "encryptionkey@0"
I20260201 03:02:40.507301 22965 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:40.524547 22965 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:40.527642 22965 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:40.527843 22965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:40.528420 22965 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:40.529336 22965 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:40.529386 22965 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:40.529438 22965 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:40.529464 22965 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:40.538426 22965 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:32873
I20260201 03:02:40.538451 23093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:32873 every 8 connection(s)
I20260201 03:02:40.539412 22965 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:02:40.542363 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 22965
I20260201 03:02:40.542490 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:02:40.542783 19247 external_mini_cluster.cc:1468] Setting key b75454ad282a9c9bf186584b60851253
Feb 01 03:02:40 dist-test-slave-gf59 krb5kdc[22853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914960, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:40.552909 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:40.541314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:42815 (local address 127.18.203.254:33943)
0201 03:02:40.541481 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:40.541485 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:40.542220 (+ 735us) server_negotiation.cc:408] Connection header received
0201 03:02:40.543324 (+ 1104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:40.543328 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:40.543385 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:40.543453 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:40.545602 (+ 2149us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:40.546121 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:40.546841 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:40.547035 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:40.549293 (+ 2258us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:40.549317 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:40.549320 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:40.549355 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:40.550893 (+ 1538us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:40.551369 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:40.551374 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:40.551376 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:40.551439 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:40.551715 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:40.551730 (+ 15us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:40.551732 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:40.551952 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:40.552065 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:40.552583 (+ 518us) server_negotiation.cc:300] Negotiation successful
0201 03:02:40.552729 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
I20260201 03:02:40.553463 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:40.541609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33943 (local address 127.18.203.193:42815)
0201 03:02:40.542053 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:40.542113 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:40.543086 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:40.543676 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:40.543687 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:40.544168 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:40.545439 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:40.545457 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:40.546242 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:40.546246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:40.546720 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:40.546729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:40.546961 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:40.547578 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:40.547604 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:40.549113 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:40.551011 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:40.551016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:40.551026 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:40.551270 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:40.551526 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:40.551528 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:40.551529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:40.551631 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:40.552052 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:40.552057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:40.552313 (+ 256us) client_negotiation.cc:769] Sending connection context
0201 03:02:40.552486 (+ 173us) client_negotiation.cc:240] Negotiation successful
0201 03:02:40.552711 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260201 03:02:40.554615 23094 heartbeater.cc:344] Connected to a master server at 127.18.203.254:33943
I20260201 03:02:40.554859 23094 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:40.555379 23094 heartbeater.cc:507] Master 127.18.203.254:33943 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
I20260201 03:02:40.556931 22901 ts_manager.cc:194] Registered new tserver with Master: e643f799b1e84cae861f7af7a1344a34 (127.18.203.193:32873)
I20260201 03:02:40.558370 22901 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:42815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:40.592191 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:33943
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:40.700367 23101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:40.700618 23101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:40.700685 23101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:40.704172 23101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:40.704249 23101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:40.704355 23101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:02:40.708618 23101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:33943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:40.709719 23101 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:40.710595 23101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:40.717300 23109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:40.717324 23106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:40.717371 23107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:40.717674 23101 server_base.cc:1060] running on GCE node
I20260201 03:02:40.718052 23101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:40.718605 23101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:40.719756 23101 hybrid_clock.cc:648] HybridClock initialized: now 1769914960719732 us; error 38 us; skew 500 ppm
Feb 01 03:02:40 dist-test-slave-gf59 krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914960, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:40.722539 23101 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:02:40.723730 23101 webserver.cc:492] Webserver started at http://127.18.203.194:44035/ using document root <none> and password file <none>
I20260201 03:02:40.724417 23101 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:40.724485 23101 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:40.724696 23101 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:40.726922 23101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "2448d755c297436eb08788cfc4d1861d"
format_stamp: "Formatted at 2026-02-01 03:02:40 on dist-test-slave-gf59"
server_key: "e5dbe55a61b0ead6421d0f48728f3ec3"
server_key_iv: "a1276b2d7a860adb24ed9c36bc9844e7"
server_key_version: "encryptionkey@0"
I20260201 03:02:40.727483 23101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "2448d755c297436eb08788cfc4d1861d"
format_stamp: "Formatted at 2026-02-01 03:02:40 on dist-test-slave-gf59"
server_key: "e5dbe55a61b0ead6421d0f48728f3ec3"
server_key_iv: "a1276b2d7a860adb24ed9c36bc9844e7"
server_key_version: "encryptionkey@0"
I20260201 03:02:40.731916 23101 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260201 03:02:40.734262 23116 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:40.735369 23101 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:40.735477 23101 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "2448d755c297436eb08788cfc4d1861d"
format_stamp: "Formatted at 2026-02-01 03:02:40 on dist-test-slave-gf59"
server_key: "e5dbe55a61b0ead6421d0f48728f3ec3"
server_key_iv: "a1276b2d7a860adb24ed9c36bc9844e7"
server_key_version: "encryptionkey@0"
I20260201 03:02:40.735569 23101 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:40.750700 23101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:40.753335 23101 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:40.753520 23101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:40.754079 23101 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:40.754967 23101 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:40.755019 23101 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:40.755071 23101 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:40.755107 23101 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:40.763747 23101 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:42163
I20260201 03:02:40.763783 23229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:42163 every 8 connection(s)
I20260201 03:02:40.764703 23101 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:02:40.767892 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 23101
I20260201 03:02:40.768013 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:02:40.768304 19247 external_mini_cluster.cc:1468] Setting key cff1cf704b9ac0fc6837256258a514e9
Feb 01 03:02:40 dist-test-slave-gf59 krb5kdc[22853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914960, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:40.777634 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:40.766319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:42667 (local address 127.18.203.254:33943)
0201 03:02:40.766494 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:40.766498 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:40.767147 (+ 649us) server_negotiation.cc:408] Connection header received
0201 03:02:40.768082 (+ 935us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:40.768086 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:40.768144 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:40.768219 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:40.770444 (+ 2225us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:40.770964 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:40.771762 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:40.771973 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:40.774381 (+ 2408us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:40.774397 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:40.774400 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:40.774426 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:40.775738 (+ 1312us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:40.776277 (+ 539us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:40.776280 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:40.776281 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:40.776321 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:40.776596 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:40.776598 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:40.776600 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:40.776738 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:40.776809 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:40.777408 (+ 599us) server_negotiation.cc:300] Negotiation successful
0201 03:02:40.777507 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
I20260201 03:02:40.778628 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:40.766598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33943 (local address 127.18.203.194:42667)
0201 03:02:40.767001 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:40.767037 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:40.767837 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:40.768407 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:40.768418 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:40.768952 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:40.770301 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:40.770319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:40.771095 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:40.771098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:40.771651 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:40.771661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:40.771886 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:40.772520 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:40.772544 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:40.774228 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:40.775853 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:40.775860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:40.775872 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:40.776186 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:40.776393 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:40.776396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:40.776398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:40.776517 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:40.776844 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:40.776851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:40.777151 (+ 300us) client_negotiation.cc:769] Sending connection context
0201 03:02:40.777393 (+ 242us) client_negotiation.cc:240] Negotiation successful
0201 03:02:40.777679 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
I20260201 03:02:40.779728 23230 heartbeater.cc:344] Connected to a master server at 127.18.203.254:33943
I20260201 03:02:40.779963 23230 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:40.780472 23230 heartbeater.cc:507] Master 127.18.203.254:33943 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
I20260201 03:02:40.781523 22901 ts_manager.cc:194] Registered new tserver with Master: 2448d755c297436eb08788cfc4d1861d (127.18.203.194:42163)
I20260201 03:02:40.782080 22901 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:42667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:40.816390 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:33943
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:40.923719 23237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:40.924017 23237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:40.924094 23237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:40.927373 23237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:40.927460 23237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:40.927563 23237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:02:40.931933 23237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:33943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:40.933061 23237 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:40.933946 23237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:40.940443 23243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:40.940428 23245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:40.940424 23242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:40.940649 23237 server_base.cc:1060] running on GCE node
I20260201 03:02:40.941073 23237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:40.941672 23237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:40.942814 23237 hybrid_clock.cc:648] HybridClock initialized: now 1769914960942799 us; error 30 us; skew 500 ppm
Feb 01 03:02:40 dist-test-slave-gf59 krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914960, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:40.945433 23237 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:02:40.946589 23237 webserver.cc:492] Webserver started at http://127.18.203.195:43521/ using document root <none> and password file <none>
I20260201 03:02:40.947126 23237 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:40.947186 23237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:40.947366 23237 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:40.949096 23237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "f579eca6c4fe48529d140f3060e69bec"
format_stamp: "Formatted at 2026-02-01 03:02:40 on dist-test-slave-gf59"
server_key: "6e843713e0f18e3f9d5cedf411aea8be"
server_key_iv: "1c2c2dc2f52005a1fa3eaaf9a45fa983"
server_key_version: "encryptionkey@0"
I20260201 03:02:40.949579 23237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "f579eca6c4fe48529d140f3060e69bec"
format_stamp: "Formatted at 2026-02-01 03:02:40 on dist-test-slave-gf59"
server_key: "6e843713e0f18e3f9d5cedf411aea8be"
server_key_iv: "1c2c2dc2f52005a1fa3eaaf9a45fa983"
server_key_version: "encryptionkey@0"
I20260201 03:02:40.953044 23237 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:02:40.955140 23252 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:40.956166 23237 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:40.956269 23237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "f579eca6c4fe48529d140f3060e69bec"
format_stamp: "Formatted at 2026-02-01 03:02:40 on dist-test-slave-gf59"
server_key: "6e843713e0f18e3f9d5cedf411aea8be"
server_key_iv: "1c2c2dc2f52005a1fa3eaaf9a45fa983"
server_key_version: "encryptionkey@0"
I20260201 03:02:40.956353 23237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:41.004902 23237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:41.008030 23237 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:41.008317 23237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:41.008983 23237 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:41.009951 23237 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:41.010004 23237 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:41.010056 23237 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:41.010092 23237 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:41.019866 23237 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:34703
I20260201 03:02:41.019908 23365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:34703 every 8 connection(s)
I20260201 03:02:41.020882 23237 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:02:41.022691 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 23237
I20260201 03:02:41.022804 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:02:41.023061 19247 external_mini_cluster.cc:1468] Setting key 44ae1d39cadba415b776c7de3b848294
Feb 01 03:02:41 dist-test-slave-gf59 krb5kdc[22853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914960, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:41.033265 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:41.022746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:35613 (local address 127.18.203.254:33943)
0201 03:02:41.022973 (+ 227us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:41.022979 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:41.023671 (+ 692us) server_negotiation.cc:408] Connection header received
0201 03:02:41.024613 (+ 942us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:41.024617 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:41.024675 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:41.024757 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:41.026542 (+ 1785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:41.027052 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:41.027707 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:41.027844 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:41.030155 (+ 2311us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:41.030171 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:41.030173 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:41.030202 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:41.031385 (+ 1183us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:41.031901 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:41.031904 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:41.031905 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:41.031951 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:41.032222 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:41.032224 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:41.032226 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:41.032375 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:41.032460 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:41.032953 (+ 493us) server_negotiation.cc:300] Negotiation successful
0201 03:02:41.033089 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260201 03:02:41.033952 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:41.023015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33943 (local address 127.18.203.195:35613)
0201 03:02:41.023499 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:41.023546 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:41.024401 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:41.024965 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:41.024974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:41.025399 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:41.026384 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:41.026400 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:41.027166 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:41.027168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:41.027569 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:41.027577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:41.027848 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:41.028440 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:41.028469 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:41.029999 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:41.031494 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:41.031500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:41.031513 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:41.031799 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:41.032026 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:41.032030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:41.032032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:41.032131 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:41.032472 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:41.032479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:41.032710 (+ 231us) client_negotiation.cc:769] Sending connection context
0201 03:02:41.032899 (+ 189us) client_negotiation.cc:240] Negotiation successful
0201 03:02:41.033159 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":118,"threads_started":1}
I20260201 03:02:41.035257 23366 heartbeater.cc:344] Connected to a master server at 127.18.203.254:33943
I20260201 03:02:41.035575 23366 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:41.036250 23366 heartbeater.cc:507] Master 127.18.203.254:33943 requested a full tablet report, sending...
I20260201 03:02:41.037328 22901 ts_manager.cc:194] Registered new tserver with Master: f579eca6c4fe48529d140f3060e69bec (127.18.203.195:34703)
I20260201 03:02:41.037994 22901 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:35613
I20260201 03:02:41.048882 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 01 03:02:41 dist-test-slave-gf59 krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914961, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Feb 01 03:02:41 dist-test-slave-gf59 krb5kdc[22853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914961, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:41.073313 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:41.066126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.18.203.254:33943)
0201 03:02:41.066276 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:41.066279 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:41.066336 (+ 57us) server_negotiation.cc:408] Connection header received
0201 03:02:41.066453 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:41.066455 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:41.066507 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:41.066592 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:41.067403 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:41.067968 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:41.068554 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:41.068674 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:41.070768 (+ 2094us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:41.070788 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:41.070800 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:41.070828 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:41.072148 (+ 1320us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:41.072477 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:41.072480 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:41.072481 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:41.072517 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:41.072658 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:41.072660 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:41.072661 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:41.072834 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:41.072933 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:41.073074 (+ 141us) server_negotiation.cc:300] Negotiation successful
0201 03:02:41.073181 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260201 03:02:41.073747 22901 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55612
I20260201 03:02:41.074680 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 22965
I20260201 03:02:41.081013 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 23101
I20260201 03:02:41.086524 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 23237
I20260201 03:02:41.091543 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 22869
2026-02-01T03:02:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3023 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:02:41 dist-test-slave-gf59 krb5kdc[23378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:02:41 dist-test-slave-gf59 krb5kdc[23378](info): set up 2 sockets
Feb 01 03:02:41 dist-test-slave-gf59 krb5kdc[23378](info): commencing operation
krb5kdc: starting...
W20260201 03:02:42.940034 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.826s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:02:43 dist-test-slave-gf59 krb5kdc[23378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:02:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:43.078186 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:37377
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:37377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:43.187794 23394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:43.188081 23394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:43.188155 23394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:43.191418 23394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:02:43.191495 23394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:43.191526 23394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:43.191550 23394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:02:43.191573 23394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:02:43.195955 23394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:37377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:37377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:43.197130 23394 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:43.198043 23394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:43.203842 23402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:43.203860 23399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:43.203842 23400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:43.204077 23394 server_base.cc:1060] running on GCE node
I20260201 03:02:43.204672 23394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:43.205631 23394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:43.206787 23394 hybrid_clock.cc:648] HybridClock initialized: now 1769914963206774 us; error 34 us; skew 500 ppm
Feb 01 03:02:43 dist-test-slave-gf59 krb5kdc[23378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914963, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:43.209349 23394 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:02:43.210460 23394 webserver.cc:492] Webserver started at http://127.18.203.254:33361/ using document root <none> and password file <none>
I20260201 03:02:43.211005 23394 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:43.211058 23394 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:43.211244 23394 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:43.213009 23394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "f9427ddb6b894fcf9e7a494026ef7a02"
format_stamp: "Formatted at 2026-02-01 03:02:43 on dist-test-slave-gf59"
server_key: "3a655163d7b3c08d3dc9e5be9e6c5193"
server_key_iv: "777bab870ebf73ba7fa20ab7c849601d"
server_key_version: "encryptionkey@0"
I20260201 03:02:43.213521 23394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "f9427ddb6b894fcf9e7a494026ef7a02"
format_stamp: "Formatted at 2026-02-01 03:02:43 on dist-test-slave-gf59"
server_key: "3a655163d7b3c08d3dc9e5be9e6c5193"
server_key_iv: "777bab870ebf73ba7fa20ab7c849601d"
server_key_version: "encryptionkey@0"
I20260201 03:02:43.216945 23394 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260201 03:02:43.219084 23409 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:43.220137 23394 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260201 03:02:43.220250 23394 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "f9427ddb6b894fcf9e7a494026ef7a02"
format_stamp: "Formatted at 2026-02-01 03:02:43 on dist-test-slave-gf59"
server_key: "3a655163d7b3c08d3dc9e5be9e6c5193"
server_key_iv: "777bab870ebf73ba7fa20ab7c849601d"
server_key_version: "encryptionkey@0"
I20260201 03:02:43.220342 23394 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:43.248665 23394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:43.255208 23394 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:43.255421 23394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:43.262737 23461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:37377 every 8 connection(s)
I20260201 03:02:43.262753 23394 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:37377
I20260201 03:02:43.263872 23394 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:02:43.264163 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 23394
I20260201 03:02:43.264261 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:02:43.264514 19247 external_mini_cluster.cc:1468] Setting key 104f7b49fd99eaa717e3cf94b4467bb9
I20260201 03:02:43.266808 23462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:43.271979 23462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02: Bootstrap starting.
Feb 01 03:02:43 dist-test-slave-gf59 krb5kdc[23378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:43.274369 23462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:43.275090 23462 log.cc:826] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:43.277012 23462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02: No bootstrap required, opened a new log
I20260201 03:02:43.277277 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:43.265716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38854 (local address 127.18.203.254:37377)
0201 03:02:43.266123 (+ 407us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:43.266134 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:43.266316 (+ 182us) server_negotiation.cc:408] Connection header received
0201 03:02:43.267467 (+ 1151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:43.267490 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:43.267837 (+ 347us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:43.268171 (+ 334us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:43.269088 (+ 917us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:43.269862 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:43.270518 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:43.270802 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:43.272912 (+ 2110us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:43.272957 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:43.272967 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:43.273001 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:43.274704 (+ 1703us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:43.275107 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:43.275112 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:43.275116 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:43.275203 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:43.275415 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:43.275419 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:43.275421 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:43.275907 (+ 486us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:43.276111 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:43.276387 (+ 276us) server_negotiation.cc:300] Negotiation successful
0201 03:02:43.276648 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":117,"threads_started":1}
I20260201 03:02:43.279934 23462 raft_consensus.cc:359] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9427ddb6b894fcf9e7a494026ef7a02" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 37377 } }
I20260201 03:02:43.280140 23462 raft_consensus.cc:385] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:43.280205 23462 raft_consensus.cc:740] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9427ddb6b894fcf9e7a494026ef7a02, State: Initialized, Role: FOLLOWER
I20260201 03:02:43.280604 23462 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9427ddb6b894fcf9e7a494026ef7a02" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 37377 } }
I20260201 03:02:43.280730 23462 raft_consensus.cc:399] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:02:43.280789 23462 raft_consensus.cc:493] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:02:43.280880 23462 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:43.281788 23462 raft_consensus.cc:515] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9427ddb6b894fcf9e7a494026ef7a02" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 37377 } }
I20260201 03:02:43.282141 23462 leader_election.cc:304] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9427ddb6b894fcf9e7a494026ef7a02; no voters:
I20260201 03:02:43.282434 23462 leader_election.cc:290] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:02:43.282547 23467 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:43.282776 23467 raft_consensus.cc:697] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [term 1 LEADER]: Becoming Leader. State: Replica: f9427ddb6b894fcf9e7a494026ef7a02, State: Running, Role: LEADER
I20260201 03:02:43.283080 23467 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9427ddb6b894fcf9e7a494026ef7a02" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 37377 } }
I20260201 03:02:43.283641 23462 sys_catalog.cc:565] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:02:43.285109 23468 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9427ddb6b894fcf9e7a494026ef7a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9427ddb6b894fcf9e7a494026ef7a02" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 37377 } } }
I20260201 03:02:43.285235 23468 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:43.285531 23469 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9427ddb6b894fcf9e7a494026ef7a02. Latest consensus state: current_term: 1 leader_uuid: "f9427ddb6b894fcf9e7a494026ef7a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9427ddb6b894fcf9e7a494026ef7a02" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 37377 } } }
I20260201 03:02:43.285624 23469 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:43.285710 23476 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:02:43.288616 23476 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:02:43.294355 23476 catalog_manager.cc:1357] Generated new cluster ID: cd627f0da1f143efb860a3b2e349c3c5
I20260201 03:02:43.294450 23476 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:02:43.312357 23476 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:02:43.313774 23476 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:02:43.322297 23476 catalog_manager.cc:6027] T 00000000000000000000000000000000 P f9427ddb6b894fcf9e7a494026ef7a02: Generated new TSK 0
I20260201 03:02:43.323060 23476 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:43.385119 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:37377
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:43.497921 23490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:43.498174 23490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:43.498256 23490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:43.501556 23490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:43.501632 23490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:43.501734 23490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:02:43.505965 23490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:37377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:43.507113 23490 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:43.508033 23490 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:43.514572 23496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:43.514618 23498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:43.514618 23495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:43.514791 23490 server_base.cc:1060] running on GCE node
I20260201 03:02:43.515266 23490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:43.515866 23490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:43.517011 23490 hybrid_clock.cc:648] HybridClock initialized: now 1769914963517001 us; error 27 us; skew 500 ppm
Feb 01 03:02:43 dist-test-slave-gf59 krb5kdc[23378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914963, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:43.519577 23490 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:02:43.520720 23490 webserver.cc:492] Webserver started at http://127.18.203.193:34393/ using document root <none> and password file <none>
I20260201 03:02:43.521298 23490 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:43.521350 23490 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:43.521538 23490 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:43.523310 23490 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "eb4000e8c2e94aa3a2e38d0cadaf07b2"
format_stamp: "Formatted at 2026-02-01 03:02:43 on dist-test-slave-gf59"
server_key: "1f8a52996beb2ef7efe2faaaba6df01e"
server_key_iv: "bc785f5083b93f65ae25f12b0c1b6a36"
server_key_version: "encryptionkey@0"
I20260201 03:02:43.523818 23490 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "eb4000e8c2e94aa3a2e38d0cadaf07b2"
format_stamp: "Formatted at 2026-02-01 03:02:43 on dist-test-slave-gf59"
server_key: "1f8a52996beb2ef7efe2faaaba6df01e"
server_key_iv: "bc785f5083b93f65ae25f12b0c1b6a36"
server_key_version: "encryptionkey@0"
I20260201 03:02:43.527266 23490 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260201 03:02:43.529479 23505 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:43.530510 23490 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260201 03:02:43.530616 23490 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "eb4000e8c2e94aa3a2e38d0cadaf07b2"
format_stamp: "Formatted at 2026-02-01 03:02:43 on dist-test-slave-gf59"
server_key: "1f8a52996beb2ef7efe2faaaba6df01e"
server_key_iv: "bc785f5083b93f65ae25f12b0c1b6a36"
server_key_version: "encryptionkey@0"
I20260201 03:02:43.530707 23490 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:43.547022 23490 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:43.550076 23490 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:43.550283 23490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:43.550850 23490 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:43.551786 23490 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:43.551836 23490 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:43.551888 23490 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:43.551916 23490 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:43.560876 23490 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:36669
I20260201 03:02:43.560904 23618 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:36669 every 8 connection(s)
I20260201 03:02:43.561863 23490 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:02:43.562052 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 23490
I20260201 03:02:43.562171 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:02:43.562463 19247 external_mini_cluster.cc:1468] Setting key 35a078b341c104ddc5c8d0809047da34
Feb 01 03:02:43 dist-test-slave-gf59 krb5kdc[23378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914963, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
I20260201 03:02:43.575009 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:43.563628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:44975 (local address 127.18.203.254:37377)
0201 03:02:43.563739 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:43.563742 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:43.564676 (+ 934us) server_negotiation.cc:408] Connection header received
0201 03:02:43.565699 (+ 1023us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:43.565703 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:43.565749 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:43.565809 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:43.567671 (+ 1862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:43.568192 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:43.568848 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:43.569003 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:43.571628 (+ 2625us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:43.571655 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:43.571658 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:43.571686 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:43.573042 (+ 1356us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:43.573521 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:43.573524 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:43.573526 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:43.573574 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:43.573791 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:43.573793 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:43.573794 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:43.573952 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:43.574030 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:43.574625 (+ 595us) server_negotiation.cc:300] Negotiation successful
0201 03:02:43.574832 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260201 03:02:43.575583 23621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:43.564028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:37377 (local address 127.18.203.193:44975)
0201 03:02:43.564520 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:43.564569 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:43.565469 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:43.565999 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:43.566010 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:43.566526 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:43.567486 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:43.567502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:43.568315 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:43.568318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:43.568740 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:43.568751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:43.568983 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:43.569602 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:43.569643 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:43.571432 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:43.573158 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:43.573162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:43.573173 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:43.573439 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:43.573643 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:43.573645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:43.573647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:43.573732 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:43.574036 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:43.574043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:43.574343 (+ 300us) client_negotiation.cc:769] Sending connection context
0201 03:02:43.574558 (+ 215us) client_negotiation.cc:240] Negotiation successful
0201 03:02:43.574833 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260201 03:02:43.576817 23619 heartbeater.cc:344] Connected to a master server at 127.18.203.254:37377
I20260201 03:02:43.577085 23619 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:43.577638 23619 heartbeater.cc:507] Master 127.18.203.254:37377 requested a full tablet report, sending...
I20260201 03:02:43.579133 23426 ts_manager.cc:194] Registered new tserver with Master: eb4000e8c2e94aa3a2e38d0cadaf07b2 (127.18.203.193:36669)
I20260201 03:02:43.580404 23426 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:44975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:43.612226 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:37377
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:43.721706 23626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:43.721971 23626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:43.722051 23626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:43.725492 23626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:43.725571 23626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:43.725672 23626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:02:43.729930 23626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:37377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:43.731055 23626 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:43.731957 23626 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:43.738504 23634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:43.738548 23631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:43.738498 23632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:43.738692 23626 server_base.cc:1060] running on GCE node
I20260201 03:02:43.739112 23626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:43.739665 23626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:43.740806 23626 hybrid_clock.cc:648] HybridClock initialized: now 1769914963740787 us; error 36 us; skew 500 ppm
Feb 01 03:02:43 dist-test-slave-gf59 krb5kdc[23378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914963, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:43.743463 23626 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:02:43.744558 23626 webserver.cc:492] Webserver started at http://127.18.203.194:42347/ using document root <none> and password file <none>
I20260201 03:02:43.745076 23626 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:43.745122 23626 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:43.745294 23626 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:43.746977 23626 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "b9b55bcad3ca49a7a7977ee387d53d2c"
format_stamp: "Formatted at 2026-02-01 03:02:43 on dist-test-slave-gf59"
server_key: "413dbabb6dfc8d6037f6a4de7b576e6e"
server_key_iv: "5615d2dc554b26c5cf857f6273fcd542"
server_key_version: "encryptionkey@0"
I20260201 03:02:43.747465 23626 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "b9b55bcad3ca49a7a7977ee387d53d2c"
format_stamp: "Formatted at 2026-02-01 03:02:43 on dist-test-slave-gf59"
server_key: "413dbabb6dfc8d6037f6a4de7b576e6e"
server_key_iv: "5615d2dc554b26c5cf857f6273fcd542"
server_key_version: "encryptionkey@0"
I20260201 03:02:43.750784 23626 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260201 03:02:43.752949 23641 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:43.753933 23626 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:43.754027 23626 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "b9b55bcad3ca49a7a7977ee387d53d2c"
format_stamp: "Formatted at 2026-02-01 03:02:43 on dist-test-slave-gf59"
server_key: "413dbabb6dfc8d6037f6a4de7b576e6e"
server_key_iv: "5615d2dc554b26c5cf857f6273fcd542"
server_key_version: "encryptionkey@0"
I20260201 03:02:43.754106 23626 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:43.777999 23626 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:43.781025 23626 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:43.781205 23626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:43.781729 23626 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:43.782624 23626 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:43.782673 23626 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:43.782735 23626 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:43.782755 23626 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:43.791767 23626 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:38617
I20260201 03:02:43.791810 23754 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:38617 every 8 connection(s)
I20260201 03:02:43.792755 23626 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:02:43.798333 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 23626
I20260201 03:02:43.798435 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:02:43.798652 19247 external_mini_cluster.cc:1468] Setting key 6b17909147d6a74a1ddc8ef4517d4444
Feb 01 03:02:43 dist-test-slave-gf59 krb5kdc[23378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914963, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:43.805212 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:43.794522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:35859 (local address 127.18.203.254:37377)
0201 03:02:43.794680 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:43.794684 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:43.795373 (+ 689us) server_negotiation.cc:408] Connection header received
0201 03:02:43.796274 (+ 901us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:43.796278 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:43.796344 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:43.796422 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:43.798126 (+ 1704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:43.798673 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:43.799388 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:43.799576 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:43.801912 (+ 2336us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:43.801930 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:43.801933 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:43.801960 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:43.803291 (+ 1331us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:43.803812 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:43.803815 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:43.803816 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:43.803864 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:43.804137 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:43.804140 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:43.804141 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:43.804303 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:43.804388 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:43.804902 (+ 514us) server_negotiation.cc:300] Negotiation successful
0201 03:02:43.805034 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260201 03:02:43.805883 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:43.794772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:37377 (local address 127.18.203.194:35859)
0201 03:02:43.795222 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:43.795262 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:43.796061 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:43.796599 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:43.796608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:43.797010 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:43.797962 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:43.797977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:43.798794 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:43.798797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:43.799251 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:43.799261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:43.799460 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:43.800087 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:43.800126 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:43.801754 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:43.803404 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:43.803409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:43.803423 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:43.803709 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:43.803944 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:43.803947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:43.803949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:43.804043 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:43.804402 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:43.804409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:43.804652 (+ 243us) client_negotiation.cc:769] Sending connection context
0201 03:02:43.804837 (+ 185us) client_negotiation.cc:240] Negotiation successful
0201 03:02:43.805124 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260201 03:02:43.807087 23755 heartbeater.cc:344] Connected to a master server at 127.18.203.254:37377
I20260201 03:02:43.807334 23755 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:43.807922 23755 heartbeater.cc:507] Master 127.18.203.254:37377 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
I20260201 03:02:43.808974 23426 ts_manager.cc:194] Registered new tserver with Master: b9b55bcad3ca49a7a7977ee387d53d2c (127.18.203.194:38617)
I20260201 03:02:43.809537 23426 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:35859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:43.848532 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:37377
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:43.956823 23762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:43.957085 23762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:43.957161 23762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:43.960536 23762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:43.960620 23762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:43.960722 23762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:02:43.965104 23762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:37377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:43.966259 23762 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:43.967124 23762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:43.973804 23768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:43.973830 23767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:43.973804 23770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:43.974136 23762 server_base.cc:1060] running on GCE node
I20260201 03:02:43.974506 23762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:43.975080 23762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:43.976310 23762 hybrid_clock.cc:648] HybridClock initialized: now 1769914963976293 us; error 36 us; skew 500 ppm
Feb 01 03:02:43 dist-test-slave-gf59 krb5kdc[23378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914963, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:43.978941 23762 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:02:43.980111 23762 webserver.cc:492] Webserver started at http://127.18.203.195:44261/ using document root <none> and password file <none>
I20260201 03:02:43.980667 23762 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:43.980720 23762 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:43.980906 23762 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:43.982662 23762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "3af36ca9f08743a4b85a7a444dec51d8"
format_stamp: "Formatted at 2026-02-01 03:02:43 on dist-test-slave-gf59"
server_key: "967e62b8c13d73cf138c00b3a32fc443"
server_key_iv: "5e27cad9ba65812e57cafb20785fd460"
server_key_version: "encryptionkey@0"
I20260201 03:02:43.983158 23762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "3af36ca9f08743a4b85a7a444dec51d8"
format_stamp: "Formatted at 2026-02-01 03:02:43 on dist-test-slave-gf59"
server_key: "967e62b8c13d73cf138c00b3a32fc443"
server_key_iv: "5e27cad9ba65812e57cafb20785fd460"
server_key_version: "encryptionkey@0"
I20260201 03:02:43.986644 23762 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260201 03:02:43.988847 23777 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:43.989871 23762 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:43.989986 23762 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "3af36ca9f08743a4b85a7a444dec51d8"
format_stamp: "Formatted at 2026-02-01 03:02:43 on dist-test-slave-gf59"
server_key: "967e62b8c13d73cf138c00b3a32fc443"
server_key_iv: "5e27cad9ba65812e57cafb20785fd460"
server_key_version: "encryptionkey@0"
I20260201 03:02:43.990077 23762 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:44.017359 23762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:44.020076 23762 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:44.020265 23762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:44.020823 23762 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:44.021725 23762 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:44.021776 23762 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:44.021828 23762 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:44.021854 23762 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:44.030905 23762 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:44923
I20260201 03:02:44.030936 23890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:44923 every 8 connection(s)
I20260201 03:02:44.031883 23762 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:02:44.034615 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 23762
I20260201 03:02:44.034732 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:02:44.035034 19247 external_mini_cluster.cc:1468] Setting key bc544892eb1759e539a62a998905ee69
Feb 01 03:02:44 dist-test-slave-gf59 krb5kdc[23378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914963, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:44.043751 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:44.033554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:51933 (local address 127.18.203.254:37377)
0201 03:02:44.033691 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:44.033695 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:44.034364 (+ 669us) server_negotiation.cc:408] Connection header received
0201 03:02:44.035435 (+ 1071us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:44.035438 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:44.035504 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:44.035579 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:44.037516 (+ 1937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:44.038029 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:44.038611 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:44.038736 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:44.040824 (+ 2088us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:44.040840 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:44.040843 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:44.040869 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:44.042162 (+ 1293us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:44.042549 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:44.042552 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:44.042553 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:44.042593 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:44.042801 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:44.042803 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:44.042804 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:44.042950 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:44.043020 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:44.043451 (+ 431us) server_negotiation.cc:300] Negotiation successful
0201 03:02:44.043591 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260201 03:02:44.044395 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:44.033791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:37377 (local address 127.18.203.195:51933)
0201 03:02:44.034209 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:44.034246 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:44.035198 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:44.035778 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:44.035787 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:44.036339 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:44.037386 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:44.037402 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:44.038121 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:44.038124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:44.038530 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:44.038536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:44.038703 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:44.039219 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:44.039244 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:44.040687 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:44.042249 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:44.042253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:44.042262 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:44.042477 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:44.042649 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:44.042651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:44.042652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:44.042741 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:44.043021 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:44.043026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:44.043249 (+ 223us) client_negotiation.cc:769] Sending connection context
0201 03:02:44.043419 (+ 170us) client_negotiation.cc:240] Negotiation successful
0201 03:02:44.043679 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":111,"threads_started":1}
I20260201 03:02:44.045403 23891 heartbeater.cc:344] Connected to a master server at 127.18.203.254:37377
I20260201 03:02:44.045672 23891 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:44.046166 23891 heartbeater.cc:507] Master 127.18.203.254:37377 requested a full tablet report, sending...
I20260201 03:02:44.047122 23426 ts_manager.cc:194] Registered new tserver with Master: 3af36ca9f08743a4b85a7a444dec51d8 (127.18.203.195:44923)
I20260201 03:02:44.047816 23426 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:51933
I20260201 03:02:44.048767 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 01 03:02:44 dist-test-slave-gf59 krb5kdc[23378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 01 03:02:44 dist-test-slave-gf59 krb5kdc[23378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.193@KRBTEST.COM
I20260201 03:02:44.073835 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:44.065955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.18.203.193:36669)
0201 03:02:44.066235 (+ 280us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:44.066242 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:44.066269 (+ 27us) server_negotiation.cc:408] Connection header received
0201 03:02:44.066343 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:44.066348 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:44.066521 (+ 173us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:44.066641 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:44.067403 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:44.068150 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:44.068775 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:44.068960 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:44.070953 (+ 1993us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:44.070979 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:44.070992 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:44.071018 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:44.072554 (+ 1536us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:44.072918 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:44.072923 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:44.072925 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:44.072976 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:44.073125 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:44.073127 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:44.073128 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:44.073303 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:44.073450 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:44.073570 (+ 120us) server_negotiation.cc:300] Negotiation successful
0201 03:02:44.073694 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":92,"threads_started":1}
Feb 01 03:02:44 dist-test-slave-gf59 krb5kdc[23378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:44.082960 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:44.076610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.18.203.254:37377)
0201 03:02:44.076748 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:44.076752 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:44.076766 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:02:44.076836 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:44.076839 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:44.076882 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:44.076946 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:44.077560 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:44.078043 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:44.078553 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:44.078673 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:44.080654 (+ 1981us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:44.080673 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:44.080676 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:44.080702 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:44.081775 (+ 1073us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:44.082102 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:44.082105 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:44.082106 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:44.082142 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:44.082293 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:44.082295 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:44.082296 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:44.082456 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:44.082553 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:44.082700 (+ 147us) server_negotiation.cc:300] Negotiation successful
0201 03:02:44.082797 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260201 03:02:44.083369 23426 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38858
I20260201 03:02:44.083946 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 23490
I20260201 03:02:44.090152 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 23626
I20260201 03:02:44.095697 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 23762
I20260201 03:02:44.100685 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 23394
2026-02-01T03:02:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3008 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:02:44 dist-test-slave-gf59 krb5kdc[23903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:02:44 dist-test-slave-gf59 krb5kdc[23903](info): set up 2 sockets
Feb 01 03:02:44 dist-test-slave-gf59 krb5kdc[23903](info): commencing operation
krb5kdc: starting...
W20260201 03:02:45.931944 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.809s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:02:46 dist-test-slave-gf59 krb5kdc[23903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:02:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:02:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:46.072333 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:44071
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:44071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:46.180742 23919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:46.181001 23919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:46.181062 23919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:46.184330 23919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:02:46.184404 23919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:46.184427 23919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:46.184450 23919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:02:46.184468 23919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:02:46.188781 23919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:44071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:44071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:46.189874 23919 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:46.190725 23919 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:46.196012 23924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:46.196044 23927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:46.196044 23925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:46.196771 23919 server_base.cc:1060] running on GCE node
I20260201 03:02:46.197321 23919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:46.198251 23919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:46.199397 23919 hybrid_clock.cc:648] HybridClock initialized: now 1769914966199378 us; error 38 us; skew 500 ppm
Feb 01 03:02:46 dist-test-slave-gf59 krb5kdc[23903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:46.201879 23919 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:02:46.202598 23919 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:46.202664 23919 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:46.202858 23919 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:46.204615 23919 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "df4771db805e44eaab4fb30068cdc034"
format_stamp: "Formatted at 2026-02-01 03:02:46 on dist-test-slave-gf59"
server_key: "31e8ab821cbb7606d0db510c0a5d1632"
server_key_iv: "d2957f1e2fb43786fb0e04d2739ddb08"
server_key_version: "encryptionkey@0"
I20260201 03:02:46.205112 23919 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "df4771db805e44eaab4fb30068cdc034"
format_stamp: "Formatted at 2026-02-01 03:02:46 on dist-test-slave-gf59"
server_key: "31e8ab821cbb7606d0db510c0a5d1632"
server_key_iv: "d2957f1e2fb43786fb0e04d2739ddb08"
server_key_version: "encryptionkey@0"
I20260201 03:02:46.208420 23919 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:02:46.210527 23933 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:46.211534 23919 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:46.211676 23919 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "df4771db805e44eaab4fb30068cdc034"
format_stamp: "Formatted at 2026-02-01 03:02:46 on dist-test-slave-gf59"
server_key: "31e8ab821cbb7606d0db510c0a5d1632"
server_key_iv: "d2957f1e2fb43786fb0e04d2739ddb08"
server_key_version: "encryptionkey@0"
I20260201 03:02:46.211767 23919 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:46.226661 23919 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:46.229439 23919 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:46.229626 23919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:46.236963 23919 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:44071
I20260201 03:02:46.236964 23985 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:44071 every 8 connection(s)
I20260201 03:02:46.237635 23919 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:02:46.240412 23986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:46.246049 23986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034: Bootstrap starting.
I20260201 03:02:46.247714 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 23919
I20260201 03:02:46.247823 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:02:46.248056 19247 external_mini_cluster.cc:1468] Setting key 1bc281a836915c2cfaf17b2620773c18
I20260201 03:02:46.248610 23986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:46.249477 23986 log.cc:826] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:46.251564 23986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034: No bootstrap required, opened a new log
I20260201 03:02:46.254693 23986 raft_consensus.cc:359] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4771db805e44eaab4fb30068cdc034" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 44071 } }
I20260201 03:02:46.254949 23986 raft_consensus.cc:385] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:46.255016 23986 raft_consensus.cc:740] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4771db805e44eaab4fb30068cdc034, State: Initialized, Role: FOLLOWER
I20260201 03:02:46.255507 23986 consensus_queue.cc:260] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4771db805e44eaab4fb30068cdc034" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 44071 } }
Feb 01 03:02:46 dist-test-slave-gf59 krb5kdc[23903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:46.255663 23986 raft_consensus.cc:399] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:02:46.255743 23986 raft_consensus.cc:493] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:02:46.255833 23986 raft_consensus.cc:3060] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:46.256690 23986 raft_consensus.cc:515] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4771db805e44eaab4fb30068cdc034" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 44071 } }
I20260201 03:02:46.257047 23986 leader_election.cc:304] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df4771db805e44eaab4fb30068cdc034; no voters:
I20260201 03:02:46.257381 23986 leader_election.cc:290] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:02:46.257480 23991 raft_consensus.cc:2804] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:46.257704 23991 raft_consensus.cc:697] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [term 1 LEADER]: Becoming Leader. State: Replica: df4771db805e44eaab4fb30068cdc034, State: Running, Role: LEADER
I20260201 03:02:46.258044 23991 consensus_queue.cc:237] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4771db805e44eaab4fb30068cdc034" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 44071 } }
I20260201 03:02:46.258693 23986 sys_catalog.cc:565] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:02:46.259483 23992 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df4771db805e44eaab4fb30068cdc034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4771db805e44eaab4fb30068cdc034" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 44071 } } }
I20260201 03:02:46.259599 23992 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:46.259876 23992 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df4771db805e44eaab4fb30068cdc034. Latest consensus state: current_term: 1 leader_uuid: "df4771db805e44eaab4fb30068cdc034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4771db805e44eaab4fb30068cdc034" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 44071 } } }
I20260201 03:02:46.259974 23992 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:46.261997 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:46.249094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.18.203.254:44071)
0201 03:02:46.249519 (+ 425us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:46.249529 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:46.249570 (+ 41us) server_negotiation.cc:408] Connection header received
0201 03:02:46.250265 (+ 695us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:46.250297 (+ 32us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:46.250600 (+ 303us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:46.250897 (+ 297us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:46.251979 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.252718 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:46.253595 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.253808 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:46.256228 (+ 2420us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:46.256249 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:46.256259 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:46.256295 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:46.258911 (+ 2616us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:46.259453 (+ 542us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:46.259458 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:46.259464 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:46.259570 (+ 106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:46.259907 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:46.259910 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:46.259912 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:46.260390 (+ 478us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:46.260602 (+ 212us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:46.260888 (+ 286us) server_negotiation.cc:300] Negotiation successful
0201 03:02:46.261198 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":131,"threads_started":1}
W20260201 03:02:46.263970 24007 catalog_manager.cc:1568] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260201 03:02:46.264040 24007 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260201 03:02:46.264101 23999 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:02:46.264856 23999 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:02:46.270309 23999 catalog_manager.cc:1357] Generated new cluster ID: caf44f47e05f4cd8861da7389ade4457
I20260201 03:02:46.270395 23999 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:02:46.289768 23999 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:02:46.291103 23999 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:02:46.301632 23999 catalog_manager.cc:6027] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034: Generated new TSK 0
I20260201 03:02:46.302551 23999 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:46.362855 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:44071
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:46.473119 24014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:46.473368 24014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:46.473439 24014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:46.476733 24014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:46.476810 24014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:46.476918 24014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:02:46.481426 24014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:44071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:46.482617 24014 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:46.483528 24014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:46.489943 24022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:46.489954 24019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:46.490049 24020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:46.490327 24014 server_base.cc:1060] running on GCE node
I20260201 03:02:46.490698 24014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:46.491281 24014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:46.492420 24014 hybrid_clock.cc:648] HybridClock initialized: now 1769914966492406 us; error 30 us; skew 500 ppm
Feb 01 03:02:46 dist-test-slave-gf59 krb5kdc[23903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:46.495028 24014 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:02:46.495743 24014 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:46.495805 24014 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:46.495968 24014 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:46.497682 24014 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "ce9f4e8bedbd401798a350abbbb0ab43"
format_stamp: "Formatted at 2026-02-01 03:02:46 on dist-test-slave-gf59"
server_key: "9745bc2cf569b61c45aeb707b5da93e7"
server_key_iv: "1b40a3f86b659bfa22c1195728cbddaa"
server_key_version: "encryptionkey@0"
I20260201 03:02:46.498121 24014 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "ce9f4e8bedbd401798a350abbbb0ab43"
format_stamp: "Formatted at 2026-02-01 03:02:46 on dist-test-slave-gf59"
server_key: "9745bc2cf569b61c45aeb707b5da93e7"
server_key_iv: "1b40a3f86b659bfa22c1195728cbddaa"
server_key_version: "encryptionkey@0"
I20260201 03:02:46.501430 24014 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:02:46.503495 24028 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:46.504516 24014 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:46.504611 24014 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "ce9f4e8bedbd401798a350abbbb0ab43"
format_stamp: "Formatted at 2026-02-01 03:02:46 on dist-test-slave-gf59"
server_key: "9745bc2cf569b61c45aeb707b5da93e7"
server_key_iv: "1b40a3f86b659bfa22c1195728cbddaa"
server_key_version: "encryptionkey@0"
I20260201 03:02:46.504688 24014 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:46.531149 24014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:46.533901 24014 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:46.534070 24014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:46.534734 24014 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:46.535718 24014 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:46.535770 24014 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:46.535809 24014 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:46.535830 24014 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:46.544543 24014 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:35095
I20260201 03:02:46.544581 24141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:35095 every 8 connection(s)
I20260201 03:02:46.545114 24014 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:02:46.549742 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 24014
I20260201 03:02:46.549860 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:02:46.550108 19247 external_mini_cluster.cc:1468] Setting key bd6f9606df439c366f849d2d9ff0b9cd
Feb 01 03:02:46 dist-test-slave-gf59 krb5kdc[23903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:46.557428 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:46.546765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:50271 (local address 127.18.203.254:44071)
0201 03:02:46.546913 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:46.546918 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:46.547660 (+ 742us) server_negotiation.cc:408] Connection header received
0201 03:02:46.548514 (+ 854us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:46.548518 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:46.548574 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:46.548648 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:46.550324 (+ 1676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.550903 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:46.551641 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.551816 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:46.553935 (+ 2119us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:46.553952 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:46.553954 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:46.553980 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:46.555506 (+ 1526us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:46.556009 (+ 503us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:46.556012 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:46.556013 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:46.556059 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:46.556335 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:46.556338 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:46.556339 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:46.556496 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:46.556582 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:46.557108 (+ 526us) server_negotiation.cc:300] Negotiation successful
0201 03:02:46.557257 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260201 03:02:46.558064 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:46.547065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:44071 (local address 127.18.203.193:50271)
0201 03:02:46.547496 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:46.547534 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:46.548331 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:46.548794 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:46.548802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:46.549200 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:46.550179 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:46.550196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.551032 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:46.551035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:46.551511 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:46.551519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.551741 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:46.552342 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:46.552368 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:46.553784 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:46.555641 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:46.555647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:46.555659 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:46.555904 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:46.556138 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:46.556142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:46.556143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:46.556240 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:46.556596 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:46.556603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:46.556849 (+ 246us) client_negotiation.cc:769] Sending connection context
0201 03:02:46.557043 (+ 194us) client_negotiation.cc:240] Negotiation successful
0201 03:02:46.557317 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":116,"threads_started":1}
I20260201 03:02:46.559270 24142 heartbeater.cc:344] Connected to a master server at 127.18.203.254:44071
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
I20260201 03:02:46.559581 24142 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:46.560144 24142 heartbeater.cc:507] Master 127.18.203.254:44071 requested a full tablet report, sending...
I20260201 03:02:46.561576 23950 ts_manager.cc:194] Registered new tserver with Master: ce9f4e8bedbd401798a350abbbb0ab43 (127.18.203.193:35095)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
I20260201 03:02:46.562875 23950 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:50271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:46.596856 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:44071
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:46.706373 24149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:46.706630 24149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:46.706693 24149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:46.710057 24149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:46.710161 24149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:46.710284 24149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:02:46.714780 24149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:44071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:46.715976 24149 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:46.716873 24149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:46.723394 24155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:46.723389 24157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:46.723393 24154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:46.723733 24149 server_base.cc:1060] running on GCE node
I20260201 03:02:46.725924 24149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:46.726531 24149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:46.727674 24149 hybrid_clock.cc:648] HybridClock initialized: now 1769914966727647 us; error 50 us; skew 500 ppm
Feb 01 03:02:46 dist-test-slave-gf59 krb5kdc[23903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:46.730654 24149 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:02:46.731395 24149 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:46.731462 24149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:46.731678 24149 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:46.733947 24149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "41242ba086ec40edb8856ecaba3f7252"
format_stamp: "Formatted at 2026-02-01 03:02:46 on dist-test-slave-gf59"
server_key: "d7e6bcf27141be2a5c7a1c3d902da3fa"
server_key_iv: "40a1c95ba91d8bd93fdc89ffe37ed8ea"
server_key_version: "encryptionkey@0"
I20260201 03:02:46.734517 24149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "41242ba086ec40edb8856ecaba3f7252"
format_stamp: "Formatted at 2026-02-01 03:02:46 on dist-test-slave-gf59"
server_key: "d7e6bcf27141be2a5c7a1c3d902da3fa"
server_key_iv: "40a1c95ba91d8bd93fdc89ffe37ed8ea"
server_key_version: "encryptionkey@0"
I20260201 03:02:46.737900 24149 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260201 03:02:46.740041 24163 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:46.741014 24149 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:02:46.741122 24149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "41242ba086ec40edb8856ecaba3f7252"
format_stamp: "Formatted at 2026-02-01 03:02:46 on dist-test-slave-gf59"
server_key: "d7e6bcf27141be2a5c7a1c3d902da3fa"
server_key_iv: "40a1c95ba91d8bd93fdc89ffe37ed8ea"
server_key_version: "encryptionkey@0"
I20260201 03:02:46.741218 24149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:46.757578 24149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:46.760566 24149 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:46.760749 24149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:46.761358 24149 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:46.762285 24149 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:46.762336 24149 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:46.762374 24149 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:46.762390 24149 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:46.770853 24149 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:44849
I20260201 03:02:46.770887 24276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:44849 every 8 connection(s)
I20260201 03:02:46.771420 24149 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:02:46.772737 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 24149
I20260201 03:02:46.772851 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:02:46.773123 19247 external_mini_cluster.cc:1468] Setting key fdcc96d85b6b940076503617ba0789d0
Feb 01 03:02:46 dist-test-slave-gf59 krb5kdc[23903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
I20260201 03:02:46.784019 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:46.773077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:47067 (local address 127.18.203.254:44071)
0201 03:02:46.773249 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:46.773254 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:46.773957 (+ 703us) server_negotiation.cc:408] Connection header received
0201 03:02:46.774978 (+ 1021us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:46.774981 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:46.775036 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:46.775115 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:46.776926 (+ 1811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.777430 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:46.778189 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.778322 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:46.780743 (+ 2421us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:46.780764 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:46.780767 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:46.780799 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:46.782104 (+ 1305us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:46.782570 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:46.782573 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:46.782575 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:46.782619 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:46.782911 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:46.782922 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:46.782924 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:46.783083 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:46.783172 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:46.783702 (+ 530us) server_negotiation.cc:300] Negotiation successful
0201 03:02:46.783830 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260201 03:02:46.784725 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:46.773374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:44071 (local address 127.18.203.194:47067)
0201 03:02:46.773813 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:46.773859 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:46.774774 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:46.775275 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:46.775283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:46.775712 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:46.776757 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:46.776775 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.777546 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:46.777550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:46.778086 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:46.778096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.778308 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:46.778912 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:46.778949 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:46.780542 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:46.782223 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:46.782229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:46.782241 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:46.782475 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:46.782708 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:46.782712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:46.782713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:46.782825 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:46.783192 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:46.783200 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:46.783437 (+ 237us) client_negotiation.cc:769] Sending connection context
0201 03:02:46.783660 (+ 223us) client_negotiation.cc:240] Negotiation successful
0201 03:02:46.783975 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
I20260201 03:02:46.786006 24277 heartbeater.cc:344] Connected to a master server at 127.18.203.254:44071
I20260201 03:02:46.786252 24277 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:46.786727 24277 heartbeater.cc:507] Master 127.18.203.254:44071 requested a full tablet report, sending...
I20260201 03:02:46.787863 23950 ts_manager.cc:194] Registered new tserver with Master: 41242ba086ec40edb8856ecaba3f7252 (127.18.203.194:44849)
I20260201 03:02:46.788470 23950 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:47067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:02:46.822197 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:44071
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:02:46.930593 24284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:46.930830 24284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:46.930908 24284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:46.934199 24284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:46.934283 24284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:46.934387 24284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:02:46.938701 24284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:44071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:46.939860 24284 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:46.940729 24284 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:46.947038 24290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:46.947091 24289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:46.947050 24292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:46.947304 24284 server_base.cc:1060] running on GCE node
I20260201 03:02:46.947743 24284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:46.948318 24284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:46.949466 24284 hybrid_clock.cc:648] HybridClock initialized: now 1769914966949454 us; error 31 us; skew 500 ppm
Feb 01 03:02:46 dist-test-slave-gf59 krb5kdc[23903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:02:46.952104 24284 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:02:46.952821 24284 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:46.952885 24284 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:46.953074 24284 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:46.954782 24284 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "5f2579ab2af54c5099806afe643e25d8"
format_stamp: "Formatted at 2026-02-01 03:02:46 on dist-test-slave-gf59"
server_key: "afad1fc1d30c0c2680f4e7c399812429"
server_key_iv: "5d29645dec4cdca84b33a53bff736769"
server_key_version: "encryptionkey@0"
I20260201 03:02:46.955272 24284 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "5f2579ab2af54c5099806afe643e25d8"
format_stamp: "Formatted at 2026-02-01 03:02:46 on dist-test-slave-gf59"
server_key: "afad1fc1d30c0c2680f4e7c399812429"
server_key_iv: "5d29645dec4cdca84b33a53bff736769"
server_key_version: "encryptionkey@0"
I20260201 03:02:46.958695 24284 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:02:46.960990 24298 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:46.962081 24284 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:02:46.962196 24284 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "5f2579ab2af54c5099806afe643e25d8"
format_stamp: "Formatted at 2026-02-01 03:02:46 on dist-test-slave-gf59"
server_key: "afad1fc1d30c0c2680f4e7c399812429"
server_key_iv: "5d29645dec4cdca84b33a53bff736769"
server_key_version: "encryptionkey@0"
I20260201 03:02:46.962296 24284 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:46.973592 24284 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:46.976769 24284 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:46.976984 24284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:46.977574 24284 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:46.978523 24284 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:46.978574 24284 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:46.978626 24284 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:46.978657 24284 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:46.987706 24284 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:45549
I20260201 03:02:46.987730 24411 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:45549 every 8 connection(s)
I20260201 03:02:46.988308 24284 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:02:46.988828 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 24284
I20260201 03:02:46.988971 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:02:46.989265 19247 external_mini_cluster.cc:1468] Setting key 858735ebf926260caadecde9b3ab0e03
Feb 01 03:02:46 dist-test-slave-gf59 krb5kdc[23903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:02:47.000514 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:46.990070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:44651 (local address 127.18.203.254:44071)
0201 03:02:46.990194 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:46.990198 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:46.991059 (+ 861us) server_negotiation.cc:408] Connection header received
0201 03:02:46.992108 (+ 1049us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:46.992111 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:46.992160 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:46.992232 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:46.993814 (+ 1582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.994313 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:46.994886 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.995018 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:46.997269 (+ 2251us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:46.997289 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:46.997292 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:46.997321 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:46.998699 (+ 1378us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:46.999143 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:46.999146 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:46.999147 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:46.999194 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:46.999451 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:46.999454 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:46.999456 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:46.999655 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:46.999759 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.000269 (+ 510us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.000377 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260201 03:02:47.001163 24414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:46.990467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:44071 (local address 127.18.203.195:44651)
0201 03:02:46.990930 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:46.990973 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:46.991927 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:46.992360 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:46.992368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:46.992750 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:46.993700 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:46.993712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.994400 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:46.994403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:46.994810 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:46.994816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:46.994969 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:46.995483 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:02:46.995507 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:46.997065 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:46.998806 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:46.998810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:46.998819 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:46.999052 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:46.999264 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:02:46.999266 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:02:46.999268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:02:46.999351 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:02:46.999754 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:46.999759 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:47.000043 (+ 284us) client_negotiation.cc:769] Sending connection context
0201 03:02:47.000235 (+ 192us) client_negotiation.cc:240] Negotiation successful
0201 03:02:47.000461 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":132,"threads_started":1}
I20260201 03:02:47.002240 24412 heartbeater.cc:344] Connected to a master server at 127.18.203.254:44071
I20260201 03:02:47.002526 24412 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:47.003093 24412 heartbeater.cc:507] Master 127.18.203.254:44071 requested a full tablet report, sending...
I20260201 03:02:47.004051 23950 ts_manager.cc:194] Registered new tserver with Master: 5f2579ab2af54c5099806afe643e25d8 (127.18.203.195:45549)
I20260201 03:02:47.004601 23950 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:44651
I20260201 03:02:47.014635 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260201 03:02:47.022727 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.016543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.18.203.254:44071)
0201 03:02:47.016674 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.016678 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.016770 (+ 92us) server_negotiation.cc:408] Connection header received
0201 03:02:47.016904 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.016907 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.016955 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.017027 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:47.017826 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.018350 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:47.018953 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.019099 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:47.020103 (+ 1004us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:47.020118 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:47.020120 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:02:47.020146 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:47.021431 (+ 1285us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:47.021718 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:47.021721 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:47.021722 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:47.021760 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:02:47.021891 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:02:47.021893 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:02:47.021894 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:02:47.022061 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:47.022179 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.022420 (+ 241us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.022534 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260201 03:02:47.025115 23950 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:02:47.027487 23950 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:02:47.039784 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.036248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:35095 (local address 127.0.0.1:54622)
0201 03:02:47.036658 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:47.036670 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:47.036753 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:47.037341 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:47.037343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:47.037377 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:47.037595 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:47.037603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.038516 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:47.038519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:47.039245 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:47.039254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.039351 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:47.039411 (+ 60us) client_negotiation.cc:769] Sending connection context
0201 03:02:47.039503 (+ 92us) client_negotiation.cc:240] Negotiation successful
0201 03:02:47.039604 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"spinlock_wait_cycles":2560,"thread_start_us":94,"threads_started":1}
I20260201 03:02:47.040267 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.035941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:45549 (local address 127.0.0.1:44434)
0201 03:02:47.036464 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:47.036477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:47.036551 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:47.037174 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:47.037180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:47.037202 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:47.037455 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:47.037465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.038940 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:47.038943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:47.039910 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:47.039921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.040023 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:47.040040 (+ 17us) client_negotiation.cc:769] Sending connection context
0201 03:02:47.040081 (+ 41us) client_negotiation.cc:240] Negotiation successful
0201 03:02:47.040136 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":105,"threads_started":1}
I20260201 03:02:47.040498 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.035859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.18.203.193:35095)
0201 03:02:47.036603 (+ 744us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.036610 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.036760 (+ 150us) server_negotiation.cc:408] Connection header received
0201 03:02:47.036861 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.036867 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.037051 (+ 184us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.037192 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:47.037712 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.038410 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:47.039564 (+ 1154us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.040114 (+ 550us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:47.040218 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.040276 (+ 58us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.040365 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":92,"threads_started":1}
I20260201 03:02:47.040843 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.035481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:44849 (local address 127.0.0.1:56282)
0201 03:02:47.036050 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:47.036090 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:47.036242 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:47.037317 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:47.037320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:47.037339 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:47.037623 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:47.037632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.038916 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:47.038920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:47.040192 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:47.040201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.040304 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:47.040320 (+ 16us) client_negotiation.cc:769] Sending connection context
0201 03:02:47.040547 (+ 227us) client_negotiation.cc:240] Negotiation successful
0201 03:02:47.040722 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"spinlock_wait_cycles":6016,"thread_start_us":100,"threads_started":1}
I20260201 03:02:47.041445 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.036287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44434 (local address 127.18.203.195:45549)
0201 03:02:47.036783 (+ 496us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.036791 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.036813 (+ 22us) server_negotiation.cc:408] Connection header received
0201 03:02:47.036884 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.036888 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.037027 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.037152 (+ 125us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:47.037592 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.038722 (+ 1130us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:47.040506 (+ 1784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.041048 (+ 542us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:47.041164 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.041233 (+ 69us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.041320 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":140,"threads_started":1}
I20260201 03:02:47.041723 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.035590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.18.203.194:44849)
0201 03:02:47.036943 (+ 1353us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.036948 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.036967 (+ 19us) server_negotiation.cc:408] Connection header received
0201 03:02:47.037036 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.037040 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.037171 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.037488 (+ 317us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:47.037746 (+ 258us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.038813 (+ 1067us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:47.040335 (+ 1522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.041386 (+ 1051us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:47.041464 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.041520 (+ 56us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.041609 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":106,"threads_started":1}
I20260201 03:02:47.043669 24076 tablet_service.cc:1505] Processing CreateTablet for tablet c31ec24598954ee99b2378341d7ef666 (DEFAULT_TABLE table=test-table [id=2d280f22e91e48d591b1f7d9c97d2fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:47.044142 24211 tablet_service.cc:1505] Processing CreateTablet for tablet c31ec24598954ee99b2378341d7ef666 (DEFAULT_TABLE table=test-table [id=2d280f22e91e48d591b1f7d9c97d2fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:47.044150 24346 tablet_service.cc:1505] Processing CreateTablet for tablet c31ec24598954ee99b2378341d7ef666 (DEFAULT_TABLE table=test-table [id=2d280f22e91e48d591b1f7d9c97d2fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:47.044761 24076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31ec24598954ee99b2378341d7ef666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:47.044924 24211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31ec24598954ee99b2378341d7ef666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:47.044965 24346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31ec24598954ee99b2378341d7ef666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:47.050340 24429 tablet_bootstrap.cc:492] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252: Bootstrap starting.
I20260201 03:02:47.051172 24431 tablet_bootstrap.cc:492] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43: Bootstrap starting.
I20260201 03:02:47.052129 24430 tablet_bootstrap.cc:492] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8: Bootstrap starting.
I20260201 03:02:47.053364 24431 tablet_bootstrap.cc:654] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:47.053362 24429 tablet_bootstrap.cc:654] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:47.053792 24430 tablet_bootstrap.cc:654] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:47.054214 24431 log.cc:826] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:47.054214 24429 log.cc:826] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:47.054426 24430 log.cc:826] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:47.056056 24429 tablet_bootstrap.cc:492] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252: No bootstrap required, opened a new log
I20260201 03:02:47.056102 24430 tablet_bootstrap.cc:492] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8: No bootstrap required, opened a new log
I20260201 03:02:47.056102 24431 tablet_bootstrap.cc:492] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43: No bootstrap required, opened a new log
I20260201 03:02:47.056246 24429 ts_tablet_manager.cc:1403] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260201 03:02:47.056270 24430 ts_tablet_manager.cc:1403] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260201 03:02:47.056270 24431 ts_tablet_manager.cc:1403] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260201 03:02:47.059489 24429 raft_consensus.cc:359] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41242ba086ec40edb8856ecaba3f7252" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 44849 } } peers { permanent_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35095 } } peers { permanent_uuid: "5f2579ab2af54c5099806afe643e25d8" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 45549 } }
I20260201 03:02:47.059749 24429 raft_consensus.cc:385] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:47.059804 24429 raft_consensus.cc:740] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41242ba086ec40edb8856ecaba3f7252, State: Initialized, Role: FOLLOWER
I20260201 03:02:47.059767 24430 raft_consensus.cc:359] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41242ba086ec40edb8856ecaba3f7252" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 44849 } } peers { permanent_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35095 } } peers { permanent_uuid: "5f2579ab2af54c5099806afe643e25d8" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 45549 } }
I20260201 03:02:47.059973 24430 raft_consensus.cc:385] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:47.060017 24430 raft_consensus.cc:740] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2579ab2af54c5099806afe643e25d8, State: Initialized, Role: FOLLOWER
I20260201 03:02:47.060379 24429 consensus_queue.cc:260] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41242ba086ec40edb8856ecaba3f7252" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 44849 } } peers { permanent_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35095 } } peers { permanent_uuid: "5f2579ab2af54c5099806afe643e25d8" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 45549 } }
I20260201 03:02:47.060443 24430 consensus_queue.cc:260] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41242ba086ec40edb8856ecaba3f7252" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 44849 } } peers { permanent_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35095 } } peers { permanent_uuid: "5f2579ab2af54c5099806afe643e25d8" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 45549 } }
I20260201 03:02:47.061286 24430 ts_tablet_manager.cc:1434] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260201 03:02:47.061296 24429 ts_tablet_manager.cc:1434] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260201 03:02:47.061668 24277 heartbeater.cc:499] Master 127.18.203.254:44071 was elected leader, sending a full tablet report...
I20260201 03:02:47.061769 24431 raft_consensus.cc:359] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41242ba086ec40edb8856ecaba3f7252" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 44849 } } peers { permanent_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35095 } } peers { permanent_uuid: "5f2579ab2af54c5099806afe643e25d8" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 45549 } }
I20260201 03:02:47.061955 24431 raft_consensus.cc:385] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:47.062001 24431 raft_consensus.cc:740] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9f4e8bedbd401798a350abbbb0ab43, State: Initialized, Role: FOLLOWER
I20260201 03:02:47.062391 24431 consensus_queue.cc:260] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41242ba086ec40edb8856ecaba3f7252" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 44849 } } peers { permanent_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35095 } } peers { permanent_uuid: "5f2579ab2af54c5099806afe643e25d8" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 45549 } }
I20260201 03:02:47.062964 24142 heartbeater.cc:499] Master 127.18.203.254:44071 was elected leader, sending a full tablet report...
I20260201 03:02:47.063194 24431 ts_tablet_manager.cc:1434] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260201 03:02:47.064009 24412 heartbeater.cc:499] Master 127.18.203.254:44071 was elected leader, sending a full tablet report...
I20260201 03:02:47.116647 24437 raft_consensus.cc:493] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:02:47.116834 24437 raft_consensus.cc:515] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41242ba086ec40edb8856ecaba3f7252" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 44849 } } peers { permanent_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35095 } } peers { permanent_uuid: "5f2579ab2af54c5099806afe643e25d8" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 45549 } }
I20260201 03:02:47.117930 24437 leader_election.cc:290] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41242ba086ec40edb8856ecaba3f7252 (127.18.203.194:44849), 5f2579ab2af54c5099806afe643e25d8 (127.18.203.195:45549)
I20260201 03:02:47.121025 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.118148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:44849 (local address 127.18.203.193:54573)
0201 03:02:47.118547 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:47.118559 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:47.118658 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:47.118909 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:47.118913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:47.118929 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:47.119222 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:47.119229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.120002 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:47.120006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:47.120600 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:47.120607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.120699 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:47.120723 (+ 24us) client_negotiation.cc:769] Sending connection context
0201 03:02:47.120769 (+ 46us) client_negotiation.cc:240] Negotiation successful
0201 03:02:47.120828 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":141,"threads_started":1}
I20260201 03:02:47.121052 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.118186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:45549 (local address 127.18.203.193:49353)
0201 03:02:47.118677 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:47.118690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:47.118791 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:47.119044 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:47.119046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:47.119061 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:02:47.119312 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:47.119320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.120046 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:47.120049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:47.120620 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:47.120627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.120746 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:47.120764 (+ 18us) client_negotiation.cc:769] Sending connection context
0201 03:02:47.120812 (+ 48us) client_negotiation.cc:240] Negotiation successful
0201 03:02:47.120868 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":134,"threads_started":1}
I20260201 03:02:47.121461 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.118272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:49353 (local address 127.18.203.195:45549)
0201 03:02:47.118403 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.118408 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.118767 (+ 359us) server_negotiation.cc:408] Connection header received
0201 03:02:47.118877 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.118881 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.118946 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.119025 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:47.119424 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.119930 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:47.120752 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.121216 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:47.121257 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.121307 (+ 50us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.121356 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260201 03:02:47.121474 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.118264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:54573 (local address 127.18.203.194:44849)
0201 03:02:47.118403 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.118407 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.118639 (+ 232us) server_negotiation.cc:408] Connection header received
0201 03:02:47.118755 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.118758 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.118810 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.118884 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:02:47.119338 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.119889 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:47.120726 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.121212 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:47.121253 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.121303 (+ 50us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.121365 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260201 03:02:47.122073 24231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c31ec24598954ee99b2378341d7ef666" candidate_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41242ba086ec40edb8856ecaba3f7252" is_pre_election: true
I20260201 03:02:47.122073 24366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c31ec24598954ee99b2378341d7ef666" candidate_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f2579ab2af54c5099806afe643e25d8" is_pre_election: true
I20260201 03:02:47.122361 24366 raft_consensus.cc:2468] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9f4e8bedbd401798a350abbbb0ab43 in term 0.
I20260201 03:02:47.122361 24231 raft_consensus.cc:2468] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9f4e8bedbd401798a350abbbb0ab43 in term 0.
I20260201 03:02:47.122870 24030 leader_election.cc:304] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f2579ab2af54c5099806afe643e25d8, ce9f4e8bedbd401798a350abbbb0ab43; no voters:
I20260201 03:02:47.123117 24437 raft_consensus.cc:2804] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:02:47.123190 24437 raft_consensus.cc:493] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:02:47.123224 24437 raft_consensus.cc:3060] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:47.124219 24437 raft_consensus.cc:515] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41242ba086ec40edb8856ecaba3f7252" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 44849 } } peers { permanent_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35095 } } peers { permanent_uuid: "5f2579ab2af54c5099806afe643e25d8" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 45549 } }
I20260201 03:02:47.124568 24437 leader_election.cc:290] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [CANDIDATE]: Term 1 election: Requested vote from peers 41242ba086ec40edb8856ecaba3f7252 (127.18.203.194:44849), 5f2579ab2af54c5099806afe643e25d8 (127.18.203.195:45549)
I20260201 03:02:47.124874 24231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c31ec24598954ee99b2378341d7ef666" candidate_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41242ba086ec40edb8856ecaba3f7252"
I20260201 03:02:47.124884 24366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c31ec24598954ee99b2378341d7ef666" candidate_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f2579ab2af54c5099806afe643e25d8"
I20260201 03:02:47.124987 24231 raft_consensus.cc:3060] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:47.124989 24366 raft_consensus.cc:3060] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:47.125888 24366 raft_consensus.cc:2468] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9f4e8bedbd401798a350abbbb0ab43 in term 1.
I20260201 03:02:47.125929 24231 raft_consensus.cc:2468] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9f4e8bedbd401798a350abbbb0ab43 in term 1.
I20260201 03:02:47.126190 24030 leader_election.cc:304] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f2579ab2af54c5099806afe643e25d8, ce9f4e8bedbd401798a350abbbb0ab43; no voters:
I20260201 03:02:47.126397 24437 raft_consensus.cc:2804] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:47.126598 24437 raft_consensus.cc:697] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [term 1 LEADER]: Becoming Leader. State: Replica: ce9f4e8bedbd401798a350abbbb0ab43, State: Running, Role: LEADER
I20260201 03:02:47.126933 24437 consensus_queue.cc:237] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41242ba086ec40edb8856ecaba3f7252" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 44849 } } peers { permanent_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35095 } } peers { permanent_uuid: "5f2579ab2af54c5099806afe643e25d8" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 45549 } }
I20260201 03:02:47.130041 23948 catalog_manager.cc:5654] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce9f4e8bedbd401798a350abbbb0ab43 (127.18.203.193). New cstate: current_term: 1 leader_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41242ba086ec40edb8856ecaba3f7252" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9f4e8bedbd401798a350abbbb0ab43" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f2579ab2af54c5099806afe643e25d8" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 45549 } health_report { overall_health: UNKNOWN } } }
I20260201 03:02:47.163671 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.160824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.18.203.193:35095)
0201 03:02:47.160975 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.160979 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.160994 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:02:47.161097 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.161100 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.161155 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.161238 (+ 83us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:02:47.161530 (+ 292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.162031 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:47.162614 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:47.162760 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:47.162857 (+ 97us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:02:47.163317 (+ 460us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:02:47.163410 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.163474 (+ 64us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.163522 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260201 03:02:47.169281 24366 raft_consensus.cc:1275] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8 [term 1 FOLLOWER]: Refusing update from remote peer ce9f4e8bedbd401798a350abbbb0ab43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:47.169281 24231 raft_consensus.cc:1275] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252 [term 1 FOLLOWER]: Refusing update from remote peer ce9f4e8bedbd401798a350abbbb0ab43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:47.169979 24437 consensus_queue.cc:1048] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f2579ab2af54c5099806afe643e25d8" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 45549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:47.170166 24440 consensus_queue.cc:1048] T c31ec24598954ee99b2378341d7ef666 P ce9f4e8bedbd401798a350abbbb0ab43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41242ba086ec40edb8856ecaba3f7252" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:47.177052 24444 mvcc.cc:204] Tried to move back new op lower bound from 7249571705518075904 to 7249571705354891264. Current Snapshot: MvccSnapshot[applied={T|T < 7249571705518075904}]
I20260201 03:02:47.177453 24445 mvcc.cc:204] Tried to move back new op lower bound from 7249571705518075904 to 7249571705354891264. Current Snapshot: MvccSnapshot[applied={T|T < 7249571705518075904}]
I20260201 03:02:47.185416 23950 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:47.185611 23950 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:47.188138 23950 catalog_manager.cc:5941] T 00000000000000000000000000000000 P df4771db805e44eaab4fb30068cdc034: Sending DeleteTablet for 3 replicas of tablet c31ec24598954ee99b2378341d7ef666
I20260201 03:02:47.188916 24076 tablet_service.cc:1552] Processing DeleteTablet for tablet c31ec24598954ee99b2378341d7ef666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:47 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:54622
I20260201 03:02:47.188925 24211 tablet_service.cc:1552] Processing DeleteTablet for tablet c31ec24598954ee99b2378341d7ef666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:47 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:56282
I20260201 03:02:47.189007 24346 tablet_service.cc:1552] Processing DeleteTablet for tablet c31ec24598954ee99b2378341d7ef666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:47 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:44434
I20260201 03:02:47.189476 24454 tablet_replica.cc:333] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252: stopping tablet replica
I20260201 03:02:47.189755 24454 raft_consensus.cc:2243] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:47.190070 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 24014
I20260201 03:02:47.190127 24454 raft_consensus.cc:2272] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:47.190685 24455 tablet_replica.cc:333] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8: stopping tablet replica
I20260201 03:02:47.190945 24455 raft_consensus.cc:2243] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:47.191228 24455 raft_consensus.cc:2272] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:47.192584 24454 ts_tablet_manager.cc:1916] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:47.194574 24454 ts_tablet_manager.cc:1929] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:02:47.194660 24454 log.cc:1199] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/c31ec24598954ee99b2378341d7ef666
I20260201 03:02:47.194958 24454 ts_tablet_manager.cc:1950] T c31ec24598954ee99b2378341d7ef666 P 41242ba086ec40edb8856ecaba3f7252: Deleting consensus metadata
I20260201 03:02:47.196094 23934 catalog_manager.cc:4985] TS 41242ba086ec40edb8856ecaba3f7252 (127.18.203.194:44849): tablet c31ec24598954ee99b2378341d7ef666 (table test-table [id=2d280f22e91e48d591b1f7d9c97d2fbd]) successfully deleted
I20260201 03:02:47.196861 24455 ts_tablet_manager.cc:1916] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260201 03:02:47.197799 23937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.203.193:35095 (error 108)
I20260201 03:02:47.197888 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 24149
I20260201 03:02:47.199115 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.198862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:35095 (local address 127.0.0.1:54640)
0201 03:02:47.198983 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:47.199039 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.193:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260201 03:02:47.199230 23937 catalog_manager.cc:4712] TS ce9f4e8bedbd401798a350abbbb0ab43 (127.18.203.193:35095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c31ec24598954ee99b2378341d7ef666: Network error: Client connection negotiation failed: client connection to 127.18.203.193:35095: connect: Connection refused (error 111)
I20260201 03:02:47.200945 24455 ts_tablet_manager.cc:1929] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:02:47.201049 24455 log.cc:1199] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/c31ec24598954ee99b2378341d7ef666
I20260201 03:02:47.201385 24455 ts_tablet_manager.cc:1950] T c31ec24598954ee99b2378341d7ef666 P 5f2579ab2af54c5099806afe643e25d8: Deleting consensus metadata
I20260201 03:02:47.202601 23935 catalog_manager.cc:4985] TS 5f2579ab2af54c5099806afe643e25d8 (127.18.203.195:45549): tablet c31ec24598954ee99b2378341d7ef666 (table test-table [id=2d280f22e91e48d591b1f7d9c97d2fbd]) successfully deleted
I20260201 03:02:47.205034 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 24284
I20260201 03:02:47.210412 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 23919
2026-02-01T03:02:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3110 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-02-01T03:02:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:02:47Z Disabled control of system clock
I20260201 03:02:47.241374 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:32935
--webserver_interface=127.18.203.254
--webserver_port=0
--builtin_ntp_servers=127.18.203.212:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:32935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260201 03:02:47.350119 24460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:47.350373 24460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:47.350426 24460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:47.353744 24460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:02:47.353823 24460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:47.353840 24460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:47.353874 24460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:02:47.353905 24460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:02:47.357884 24460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:32935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:32935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:47.358976 24460 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:47.360150 24460 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:47.365700 24468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:47.365694 24465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:47.365694 24466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:47.366005 24460 server_base.cc:1060] running on GCE node
I20260201 03:02:47.366621 24460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:47.367524 24460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:47.368672 24460 hybrid_clock.cc:648] HybridClock initialized: now 1769914967368663 us; error 26 us; skew 500 ppm
I20260201 03:02:47.370514 24460 webserver.cc:492] Webserver started at http://127.18.203.254:38249/ using document root <none> and password file <none>
I20260201 03:02:47.371079 24460 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:47.371138 24460 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:47.371331 24460 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:47.373066 24460 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "68d87e4571744a80b24cbc771aef5fd6"
format_stamp: "Formatted at 2026-02-01 03:02:47 on dist-test-slave-gf59"
server_key: "b727731260aee713252f2f82945178f6"
server_key_iv: "dc92002dbbafb2ef9651d65bc9045e06"
server_key_version: "encryptionkey@0"
I20260201 03:02:47.373528 24460 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "68d87e4571744a80b24cbc771aef5fd6"
format_stamp: "Formatted at 2026-02-01 03:02:47 on dist-test-slave-gf59"
server_key: "b727731260aee713252f2f82945178f6"
server_key_iv: "dc92002dbbafb2ef9651d65bc9045e06"
server_key_version: "encryptionkey@0"
I20260201 03:02:47.376875 24460 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:02:47.379099 24474 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:47.380115 24460 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:02:47.380232 24460 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "68d87e4571744a80b24cbc771aef5fd6"
format_stamp: "Formatted at 2026-02-01 03:02:47 on dist-test-slave-gf59"
server_key: "b727731260aee713252f2f82945178f6"
server_key_iv: "dc92002dbbafb2ef9651d65bc9045e06"
server_key_version: "encryptionkey@0"
I20260201 03:02:47.380318 24460 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:47.385418 24460 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:47.385964 24460 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:47.386113 24460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:47.393029 24526 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:32935 every 8 connection(s)
I20260201 03:02:47.393034 24460 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:32935
I20260201 03:02:47.394126 24460 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:02:47.396930 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 24460
I20260201 03:02:47.396961 24527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:47.397053 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:02:47.397302 19247 external_mini_cluster.cc:1468] Setting key 9d0d59384a84cd390f0505a8be7b52dc
I20260201 03:02:47.403301 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.398581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.18.203.254:32935)
0201 03:02:47.399298 (+ 717us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.399310 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.399357 (+ 47us) server_negotiation.cc:408] Connection header received
0201 03:02:47.400428 (+ 1071us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.400458 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.400880 (+ 422us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.401273 (+ 393us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:47.401953 (+ 680us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:47.401959 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:47.401970 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:47.401981 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:47.402028 (+ 47us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:47.402044 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:47.402198 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.402290 (+ 92us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.402525 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":106,"threads_started":1}
I20260201 03:02:47.404142 24527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6: Bootstrap starting.
I20260201 03:02:47.406332 24527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:47.407009 24527 log.cc:826] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:47.408836 24527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6: No bootstrap required, opened a new log
I20260201 03:02:47.411439 24527 raft_consensus.cc:359] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d87e4571744a80b24cbc771aef5fd6" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 32935 } }
I20260201 03:02:47.411661 24527 raft_consensus.cc:385] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:47.411718 24527 raft_consensus.cc:740] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d87e4571744a80b24cbc771aef5fd6, State: Initialized, Role: FOLLOWER
I20260201 03:02:47.412097 24527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d87e4571744a80b24cbc771aef5fd6" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 32935 } }
I20260201 03:02:47.412214 24527 raft_consensus.cc:399] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:02:47.412259 24527 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:02:47.412324 24527 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:47.413151 24527 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d87e4571744a80b24cbc771aef5fd6" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 32935 } }
I20260201 03:02:47.413480 24527 leader_election.cc:304] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68d87e4571744a80b24cbc771aef5fd6; no voters:
I20260201 03:02:47.413772 24527 leader_election.cc:290] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:02:47.413874 24532 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:47.414073 24532 raft_consensus.cc:697] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [term 1 LEADER]: Becoming Leader. State: Replica: 68d87e4571744a80b24cbc771aef5fd6, State: Running, Role: LEADER
I20260201 03:02:47.414387 24532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d87e4571744a80b24cbc771aef5fd6" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 32935 } }
I20260201 03:02:47.414909 24527 sys_catalog.cc:565] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:02:47.416106 24534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68d87e4571744a80b24cbc771aef5fd6. Latest consensus state: current_term: 1 leader_uuid: "68d87e4571744a80b24cbc771aef5fd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d87e4571744a80b24cbc771aef5fd6" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 32935 } } }
I20260201 03:02:47.416122 24533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68d87e4571744a80b24cbc771aef5fd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d87e4571744a80b24cbc771aef5fd6" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 32935 } } }
I20260201 03:02:47.416244 24533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:47.416244 24534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6 [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:47.416766 24541 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:02:47.419430 24541 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:02:47.424927 24541 catalog_manager.cc:1357] Generated new cluster ID: 50436edce2a14e18a1b5c43f802d338b
I20260201 03:02:47.425021 24541 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:02:47.438126 24541 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:02:47.438501 24541 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:02:47.447222 24541 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6: Generated new TSK 0
I20260201 03:02:47.448146 24541 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:02:47.458920 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:32935
--builtin_ntp_servers=127.18.203.212:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260201 03:02:47.570013 24551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:47.570282 24551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:47.570355 24551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:47.573791 24551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:47.573879 24551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:47.573979 24551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:02:47.577979 24551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:32935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:47.579149 24551 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:47.580358 24551 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:47.587249 24557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:47.587240 24556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:47.587237 24559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:47.587519 24551 server_base.cc:1060] running on GCE node
I20260201 03:02:47.587899 24551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:47.588497 24551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:47.589651 24551 hybrid_clock.cc:648] HybridClock initialized: now 1769914967589630 us; error 37 us; skew 500 ppm
I20260201 03:02:47.591487 24551 webserver.cc:492] Webserver started at http://127.18.203.193:46421/ using document root <none> and password file <none>
I20260201 03:02:47.592092 24551 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:47.592157 24551 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:47.592362 24551 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:47.594082 24551 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "9a5e47ca4c894c898c91db64b4029ca6"
format_stamp: "Formatted at 2026-02-01 03:02:47 on dist-test-slave-gf59"
server_key: "135a6d833d7fa913ccf5baa8a792dbd7"
server_key_iv: "7d1969be1d63b5676f795d335321e1e9"
server_key_version: "encryptionkey@0"
I20260201 03:02:47.594571 24551 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "9a5e47ca4c894c898c91db64b4029ca6"
format_stamp: "Formatted at 2026-02-01 03:02:47 on dist-test-slave-gf59"
server_key: "135a6d833d7fa913ccf5baa8a792dbd7"
server_key_iv: "7d1969be1d63b5676f795d335321e1e9"
server_key_version: "encryptionkey@0"
I20260201 03:02:47.598027 24551 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:02:47.600371 24565 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:47.601426 24551 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:02:47.601541 24551 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "9a5e47ca4c894c898c91db64b4029ca6"
format_stamp: "Formatted at 2026-02-01 03:02:47 on dist-test-slave-gf59"
server_key: "135a6d833d7fa913ccf5baa8a792dbd7"
server_key_iv: "7d1969be1d63b5676f795d335321e1e9"
server_key_version: "encryptionkey@0"
I20260201 03:02:47.601639 24551 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:47.606624 24551 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:47.607259 24551 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:47.607434 24551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:47.607997 24551 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:47.608990 24551 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:47.609040 24551 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:47.609092 24551 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:47.609120 24551 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:47.617890 24551 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:40941
I20260201 03:02:47.617916 24678 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:40941 every 8 connection(s)
I20260201 03:02:47.618892 24551 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:02:47.624228 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.620695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:39365 (local address 127.18.203.254:32935)
0201 03:02:47.620843 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.620847 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.621758 (+ 911us) server_negotiation.cc:408] Connection header received
0201 03:02:47.622563 (+ 805us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.622567 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.622616 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.622711 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:47.623740 (+ 1029us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:47.623744 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:47.623747 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:47.623762 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:47.623784 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:47.623792 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:47.623870 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.624030 (+ 160us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.624077 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260201 03:02:47.624905 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.620981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:32935 (local address 127.18.203.193:39365)
0201 03:02:47.621622 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:47.621657 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:47.622366 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:47.622872 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:47.622880 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:47.622978 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:47.623485 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:02:47.623487 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:47.623520 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:02:47.623573 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:47.623873 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:47.623882 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:47.623888 (+ 6us) client_negotiation.cc:769] Sending connection context
0201 03:02:47.624089 (+ 201us) client_negotiation.cc:240] Negotiation successful
0201 03:02:47.624300 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":117,"threads_started":1}
I20260201 03:02:47.625861 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 24551
I20260201 03:02:47.625967 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:02:47.625990 24679 heartbeater.cc:344] Connected to a master server at 127.18.203.254:32935
I20260201 03:02:47.626250 24679 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:47.626272 19247 external_mini_cluster.cc:1468] Setting key 397047a917558339e6df90828db8f1fd
I20260201 03:02:47.626710 24679 heartbeater.cc:507] Master 127.18.203.254:32935 requested a full tablet report, sending...
I20260201 03:02:47.628149 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:32935
--builtin_ntp_servers=127.18.203.212:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260201 03:02:47.628229 24491 ts_manager.cc:194] Registered new tserver with Master: 9a5e47ca4c894c898c91db64b4029ca6 (127.18.203.193:40941)
W20260201 03:02:47.739792 24682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:47.740063 24682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:47.740118 24682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:47.743407 24682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:47.743475 24682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:47.743551 24682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:02:47.748020 24682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:32935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:47.749188 24682 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:47.750300 24682 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:47.756887 24687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:47.756933 24688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:47.757017 24690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:47.757092 24682 server_base.cc:1060] running on GCE node
I20260201 03:02:47.757560 24682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:47.758124 24682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:47.759272 24682 hybrid_clock.cc:648] HybridClock initialized: now 1769914967759259 us; error 30 us; skew 500 ppm
I20260201 03:02:47.761149 24682 webserver.cc:492] Webserver started at http://127.18.203.194:38751/ using document root <none> and password file <none>
I20260201 03:02:47.761723 24682 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:47.761785 24682 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:47.761971 24682 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:47.763785 24682 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "de95131e07f04a5ab7724dadde2c4918"
format_stamp: "Formatted at 2026-02-01 03:02:47 on dist-test-slave-gf59"
server_key: "544c7803c01eca748a157ac5611ad766"
server_key_iv: "f576754969aa02bef4ff72260e598577"
server_key_version: "encryptionkey@0"
I20260201 03:02:47.764279 24682 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "de95131e07f04a5ab7724dadde2c4918"
format_stamp: "Formatted at 2026-02-01 03:02:47 on dist-test-slave-gf59"
server_key: "544c7803c01eca748a157ac5611ad766"
server_key_iv: "f576754969aa02bef4ff72260e598577"
server_key_version: "encryptionkey@0"
I20260201 03:02:47.767719 24682 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:02:47.770038 24696 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:47.771116 24682 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:47.771234 24682 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "de95131e07f04a5ab7724dadde2c4918"
format_stamp: "Formatted at 2026-02-01 03:02:47 on dist-test-slave-gf59"
server_key: "544c7803c01eca748a157ac5611ad766"
server_key_iv: "f576754969aa02bef4ff72260e598577"
server_key_version: "encryptionkey@0"
I20260201 03:02:47.771334 24682 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:47.776544 24682 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:47.777190 24682 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:47.777375 24682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:47.777923 24682 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:47.778967 24682 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:47.779023 24682 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:47.779078 24682 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:47.779119 24682 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:47.788533 24682 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:34847
I20260201 03:02:47.788561 24809 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:34847 every 8 connection(s)
I20260201 03:02:47.789590 24682 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:02:47.794981 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.791411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:44043 (local address 127.18.203.254:32935)
0201 03:02:47.791595 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.791600 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.792607 (+ 1007us) server_negotiation.cc:408] Connection header received
0201 03:02:47.793377 (+ 770us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.793380 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.793427 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.793517 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:47.794568 (+ 1051us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:47.794572 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:47.794574 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:47.794588 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:47.794611 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:47.794619 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:47.794706 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.794815 (+ 109us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.794855 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260201 03:02:47.795190 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 24682
I20260201 03:02:47.795323 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:02:47.795635 19247 external_mini_cluster.cc:1468] Setting key 7e665229ea34e05ea03f50ef4b30fd4c
I20260201 03:02:47.795936 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.791770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:32935 (local address 127.18.203.194:44043)
0201 03:02:47.792465 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:47.792501 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:47.793183 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:47.793665 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:47.793673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:47.793759 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:47.794320 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:02:47.794322 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:47.794357 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:02:47.794415 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:47.794695 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:47.794700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:47.794704 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:02:47.794866 (+ 162us) client_negotiation.cc:240] Negotiation successful
0201 03:02:47.795098 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":105,"threads_started":1}
I20260201 03:02:47.797024 24810 heartbeater.cc:344] Connected to a master server at 127.18.203.254:32935
I20260201 03:02:47.797300 24810 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:47.797762 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:32935
--builtin_ntp_servers=127.18.203.212:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260201 03:02:47.797778 24810 heartbeater.cc:507] Master 127.18.203.254:32935 requested a full tablet report, sending...
I20260201 03:02:47.798869 24491 ts_manager.cc:194] Registered new tserver with Master: de95131e07f04a5ab7724dadde2c4918 (127.18.203.194:34847)
W20260201 03:02:47.908237 24813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:47.908514 24813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:47.908593 24813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:47.911890 24813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:47.911967 24813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:47.912070 24813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:02:47.916317 24813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:32935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:47.917512 24813 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:47.918648 24813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:47.925395 24821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:47.925403 24819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:47.925426 24818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:47.925750 24813 server_base.cc:1060] running on GCE node
I20260201 03:02:47.926113 24813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:47.926666 24813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:47.927802 24813 hybrid_clock.cc:648] HybridClock initialized: now 1769914967927786 us; error 35 us; skew 500 ppm
I20260201 03:02:47.929566 24813 webserver.cc:492] Webserver started at http://127.18.203.195:34247/ using document root <none> and password file <none>
I20260201 03:02:47.930128 24813 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:47.930187 24813 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:47.930367 24813 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:47.932112 24813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "41d93deaa6b5407cafbf961bfcf8665b"
format_stamp: "Formatted at 2026-02-01 03:02:47 on dist-test-slave-gf59"
server_key: "1a02909a5dbaef976481ba0a16597391"
server_key_iv: "bb76df9898500d699faabb67e903e361"
server_key_version: "encryptionkey@0"
I20260201 03:02:47.932587 24813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "41d93deaa6b5407cafbf961bfcf8665b"
format_stamp: "Formatted at 2026-02-01 03:02:47 on dist-test-slave-gf59"
server_key: "1a02909a5dbaef976481ba0a16597391"
server_key_iv: "bb76df9898500d699faabb67e903e361"
server_key_version: "encryptionkey@0"
I20260201 03:02:47.935962 24813 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:02:47.938112 24827 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:47.939129 24813 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:47.939235 24813 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "41d93deaa6b5407cafbf961bfcf8665b"
format_stamp: "Formatted at 2026-02-01 03:02:47 on dist-test-slave-gf59"
server_key: "1a02909a5dbaef976481ba0a16597391"
server_key_iv: "bb76df9898500d699faabb67e903e361"
server_key_version: "encryptionkey@0"
I20260201 03:02:47.939329 24813 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:47.944308 24813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:47.944911 24813 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:47.945084 24813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:47.945601 24813 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:02:47.946602 24813 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:02:47.946650 24813 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:47.946699 24813 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:02:47.946727 24813 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:47.955340 24813 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:33547
I20260201 03:02:47.955368 24940 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:33547 every 8 connection(s)
I20260201 03:02:47.956410 24813 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:02:47.961571 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.958155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:38273 (local address 127.18.203.254:32935)
0201 03:02:47.958330 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.958335 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.959233 (+ 898us) server_negotiation.cc:408] Connection header received
0201 03:02:47.960068 (+ 835us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.960071 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.960113 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.960196 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:47.961177 (+ 981us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:47.961180 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:47.961183 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:47.961194 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:47.961216 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:47.961224 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:47.961301 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.961420 (+ 119us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.961457 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260201 03:02:47.962333 24943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.958470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:32935 (local address 127.18.203.195:38273)
0201 03:02:47.959095 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:47.959132 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:47.959886 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:47.960346 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:47.960354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:47.960440 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:47.960946 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:02:47.960948 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:47.960981 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:02:47.961037 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:47.961289 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:47.961294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:47.961298 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:02:47.961477 (+ 179us) client_negotiation.cc:240] Negotiation successful
0201 03:02:47.961666 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
I20260201 03:02:47.963385 24941 heartbeater.cc:344] Connected to a master server at 127.18.203.254:32935
I20260201 03:02:47.963656 24941 heartbeater.cc:461] Registering TS with master...
I20260201 03:02:47.963958 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 24813
I20260201 03:02:47.964053 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:02:47.964105 24941 heartbeater.cc:507] Master 127.18.203.254:32935 requested a full tablet report, sending...
I20260201 03:02:47.964329 19247 external_mini_cluster.cc:1468] Setting key 3028bab07790c5bd4eab90203c7359bb
I20260201 03:02:47.965153 24490 ts_manager.cc:194] Registered new tserver with Master: 41d93deaa6b5407cafbf961bfcf8665b (127.18.203.195:33547)
I20260201 03:02:47.966354 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260201 03:02:47.969509 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.968119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33928 (local address 127.18.203.254:32935)
0201 03:02:47.968252 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.968256 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.968402 (+ 146us) server_negotiation.cc:408] Connection header received
0201 03:02:47.968519 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.968521 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.968554 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.968611 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:47.969209 (+ 598us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:47.969212 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:47.969214 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:47.969224 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:47.969245 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:47.969252 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:47.969318 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.969373 (+ 55us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.969404 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260201 03:02:47.972069 24490 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:02:47.974566 24490 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:02:47.985519 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.982541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:34847 (local address 127.0.0.1:42294)
0201 03:02:47.982984 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:47.983028 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:47.983132 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:47.984026 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:47.984033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:47.984101 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:47.984675 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:02:47.984677 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:47.984718 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:02:47.984778 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:47.985102 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:47.985155 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:47.985161 (+ 6us) client_negotiation.cc:769] Sending connection context
0201 03:02:47.985294 (+ 133us) client_negotiation.cc:240] Negotiation successful
0201 03:02:47.985353 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":88,"threads_started":1}
I20260201 03:02:47.985519 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.982691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42294 (local address 127.18.203.194:34847)
0201 03:02:47.983541 (+ 850us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.983549 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.983576 (+ 27us) server_negotiation.cc:408] Connection header received
0201 03:02:47.983665 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.983672 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.983859 (+ 187us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.984017 (+ 158us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:47.984899 (+ 882us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:47.984906 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:47.984916 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:47.984929 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:47.984997 (+ 68us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:47.985012 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:47.985153 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.985319 (+ 166us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.985391 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":131,"threads_started":1}
I20260201 03:02:47.986413 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.983519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:40941 (local address 127.0.0.1:35066)
0201 03:02:47.983777 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:47.983790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:47.983864 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:47.984520 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:47.984524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:47.984566 (+ 42us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:47.985697 (+ 1131us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:02:47.985699 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:47.985711 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:02:47.985735 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:47.986148 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:47.986152 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:47.986154 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:02:47.986239 (+ 85us) client_negotiation.cc:240] Negotiation successful
0201 03:02:47.986272 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":120,"threads_started":1}
I20260201 03:02:47.986486 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.983699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35066 (local address 127.18.203.193:40941)
0201 03:02:47.984104 (+ 405us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.984110 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.984132 (+ 22us) server_negotiation.cc:408] Connection header received
0201 03:02:47.984187 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.984192 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.984346 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.984483 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:47.985882 (+ 1399us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:47.985889 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:47.985898 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:47.985909 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:47.985968 (+ 59us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:47.985981 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:47.986075 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.986275 (+ 200us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.986341 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":102,"threads_started":1}
I20260201 03:02:47.987015 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.983138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:33547 (local address 127.0.0.1:36772)
0201 03:02:47.983388 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:47.983399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:47.983459 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:47.985592 (+ 2133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:47.985595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:47.985609 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:47.986024 (+ 415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:02:47.986026 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:47.986039 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:02:47.986063 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:47.986788 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:47.986791 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:47.986794 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:02:47.986867 (+ 73us) client_negotiation.cc:240] Negotiation successful
0201 03:02:47.986898 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":115,"threads_started":1}
I20260201 03:02:47.987068 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:47.983824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.18.203.195:33547)
0201 03:02:47.985222 (+ 1398us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:47.985228 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:47.985249 (+ 21us) server_negotiation.cc:408] Connection header received
0201 03:02:47.985304 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:47.985309 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:47.985445 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:47.985582 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:47.986625 (+ 1043us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:47.986630 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:47.986638 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:47.986649 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:47.986690 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:47.986701 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:47.986808 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:47.986873 (+ 65us) server_negotiation.cc:300] Negotiation successful
0201 03:02:47.986931 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":159,"threads_started":1}
I20260201 03:02:47.989063 24744 tablet_service.cc:1505] Processing CreateTablet for tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b (DEFAULT_TABLE table=test-table [id=b347c210cdba4d52b3f12d340bfca9cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:47.989979 24744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:47.989665 24875 tablet_service.cc:1505] Processing CreateTablet for tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b (DEFAULT_TABLE table=test-table [id=b347c210cdba4d52b3f12d340bfca9cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:47.989670 24613 tablet_service.cc:1505] Processing CreateTablet for tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b (DEFAULT_TABLE table=test-table [id=b347c210cdba4d52b3f12d340bfca9cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:02:47.990590 24613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:47.990592 24875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:47.996210 24958 tablet_bootstrap.cc:492] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918: Bootstrap starting.
I20260201 03:02:47.997231 24960 tablet_bootstrap.cc:492] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b: Bootstrap starting.
I20260201 03:02:47.997231 24959 tablet_bootstrap.cc:492] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6: Bootstrap starting.
I20260201 03:02:47.998379 24958 tablet_bootstrap.cc:654] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:47.999110 24958 log.cc:826] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:47.999310 24959 tablet_bootstrap.cc:654] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:47.999310 24960 tablet_bootstrap.cc:654] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:48.000151 24960 log.cc:826] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:48.000152 24959 log.cc:826] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:48.001063 24958 tablet_bootstrap.cc:492] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918: No bootstrap required, opened a new log
I20260201 03:02:48.001255 24958 ts_tablet_manager.cc:1403] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260201 03:02:48.001875 24960 tablet_bootstrap.cc:492] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b: No bootstrap required, opened a new log
I20260201 03:02:48.002051 24960 ts_tablet_manager.cc:1403] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260201 03:02:48.002310 24959 tablet_bootstrap.cc:492] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6: No bootstrap required, opened a new log
I20260201 03:02:48.002489 24959 ts_tablet_manager.cc:1403] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260201 03:02:48.004442 24960 raft_consensus.cc:359] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de95131e07f04a5ab7724dadde2c4918" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 34847 } } peers { permanent_uuid: "9a5e47ca4c894c898c91db64b4029ca6" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 40941 } } peers { permanent_uuid: "41d93deaa6b5407cafbf961bfcf8665b" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33547 } }
I20260201 03:02:48.004462 24958 raft_consensus.cc:359] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de95131e07f04a5ab7724dadde2c4918" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 34847 } } peers { permanent_uuid: "9a5e47ca4c894c898c91db64b4029ca6" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 40941 } } peers { permanent_uuid: "41d93deaa6b5407cafbf961bfcf8665b" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33547 } }
I20260201 03:02:48.004635 24960 raft_consensus.cc:385] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:48.004662 24958 raft_consensus.cc:385] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:48.004715 24960 raft_consensus.cc:740] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d93deaa6b5407cafbf961bfcf8665b, State: Initialized, Role: FOLLOWER
I20260201 03:02:48.004747 24958 raft_consensus.cc:740] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de95131e07f04a5ab7724dadde2c4918, State: Initialized, Role: FOLLOWER
I20260201 03:02:48.005139 24960 consensus_queue.cc:260] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de95131e07f04a5ab7724dadde2c4918" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 34847 } } peers { permanent_uuid: "9a5e47ca4c894c898c91db64b4029ca6" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 40941 } } peers { permanent_uuid: "41d93deaa6b5407cafbf961bfcf8665b" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33547 } }
I20260201 03:02:48.005227 24958 consensus_queue.cc:260] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de95131e07f04a5ab7724dadde2c4918" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 34847 } } peers { permanent_uuid: "9a5e47ca4c894c898c91db64b4029ca6" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 40941 } } peers { permanent_uuid: "41d93deaa6b5407cafbf961bfcf8665b" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33547 } }
I20260201 03:02:48.005447 24959 raft_consensus.cc:359] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de95131e07f04a5ab7724dadde2c4918" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 34847 } } peers { permanent_uuid: "9a5e47ca4c894c898c91db64b4029ca6" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 40941 } } peers { permanent_uuid: "41d93deaa6b5407cafbf961bfcf8665b" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33547 } }
I20260201 03:02:48.005667 24959 raft_consensus.cc:385] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:48.005735 24959 raft_consensus.cc:740] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5e47ca4c894c898c91db64b4029ca6, State: Initialized, Role: FOLLOWER
I20260201 03:02:48.006132 24960 ts_tablet_manager.cc:1434] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20260201 03:02:48.006090 24958 ts_tablet_manager.cc:1434] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260201 03:02:48.006203 24959 consensus_queue.cc:260] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de95131e07f04a5ab7724dadde2c4918" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 34847 } } peers { permanent_uuid: "9a5e47ca4c894c898c91db64b4029ca6" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 40941 } } peers { permanent_uuid: "41d93deaa6b5407cafbf961bfcf8665b" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33547 } }
I20260201 03:02:48.006496 24810 heartbeater.cc:499] Master 127.18.203.254:32935 was elected leader, sending a full tablet report...
I20260201 03:02:48.006495 24941 heartbeater.cc:499] Master 127.18.203.254:32935 was elected leader, sending a full tablet report...
I20260201 03:02:48.006965 24959 ts_tablet_manager.cc:1434] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260201 03:02:48.007282 24679 heartbeater.cc:499] Master 127.18.203.254:32935 was elected leader, sending a full tablet report...
W20260201 03:02:48.040479 24811 tablet.cc:2378] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260201 03:02:48.119940 24680 tablet.cc:2378] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:02:48.141736 24966 raft_consensus.cc:493] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:02:48.141950 24966 raft_consensus.cc:515] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de95131e07f04a5ab7724dadde2c4918" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 34847 } } peers { permanent_uuid: "9a5e47ca4c894c898c91db64b4029ca6" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 40941 } } peers { permanent_uuid: "41d93deaa6b5407cafbf961bfcf8665b" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33547 } }
I20260201 03:02:48.143119 24966 leader_election.cc:290] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de95131e07f04a5ab7724dadde2c4918 (127.18.203.194:34847), 41d93deaa6b5407cafbf961bfcf8665b (127.18.203.195:33547)
I20260201 03:02:48.145078 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:48.143371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:34847 (local address 127.18.203.193:43443)
0201 03:02:48.143677 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:48.143692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:48.143808 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:48.144047 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:48.144050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:48.144065 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:48.144549 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:02:48.144552 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:48.144565 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:02:48.144593 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:48.144856 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:48.144858 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:48.144860 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:02:48.144909 (+ 49us) client_negotiation.cc:240] Negotiation successful
0201 03:02:48.144949 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":100,"threads_started":1}
I20260201 03:02:48.145123 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:48.143478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:43443 (local address 127.18.203.194:34847)
0201 03:02:48.143642 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:48.143646 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:48.143777 (+ 131us) server_negotiation.cc:408] Connection header received
0201 03:02:48.143904 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:48.143907 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:48.143945 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:48.144015 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:48.144725 (+ 710us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:48.144730 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:48.144732 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:48.144745 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:48.144767 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:48.144776 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:48.144874 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:48.144918 (+ 44us) server_negotiation.cc:300] Negotiation successful
0201 03:02:48.144978 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260201 03:02:48.145798 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:48.143502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:34415 (local address 127.18.203.195:33547)
0201 03:02:48.143676 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:48.143680 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:48.143986 (+ 306us) server_negotiation.cc:408] Connection header received
0201 03:02:48.144087 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:48.144090 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:48.144129 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:48.144197 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:48.145442 (+ 1245us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:48.145447 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:48.145450 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:48.145463 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:48.145487 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:48.145497 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:48.145582 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:48.145672 (+ 90us) server_negotiation.cc:300] Negotiation successful
0201 03:02:48.145707 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260201 03:02:48.145879 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:48.143371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:33547 (local address 127.18.203.193:34415)
0201 03:02:48.143892 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:48.143904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:48.143997 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:48.144239 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:48.144242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:48.144276 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:02:48.145285 (+ 1009us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:02:48.145287 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:02:48.145298 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:02:48.145323 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:02:48.145591 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:02:48.145594 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:02:48.145596 (+ 2us) client_negotiation.cc:769] Sending connection context
0201 03:02:48.145677 (+ 81us) client_negotiation.cc:240] Negotiation successful
0201 03:02:48.145712 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":285,"thread_start_us":72,"threads_started":1}
I20260201 03:02:48.145714 24764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f3fed0cd0d04a6e97a5fba9d43dcf8b" candidate_uuid: "9a5e47ca4c894c898c91db64b4029ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de95131e07f04a5ab7724dadde2c4918" is_pre_election: true
I20260201 03:02:48.146481 24764 raft_consensus.cc:2468] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a5e47ca4c894c898c91db64b4029ca6 in term 0.
I20260201 03:02:48.146473 24895 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f3fed0cd0d04a6e97a5fba9d43dcf8b" candidate_uuid: "9a5e47ca4c894c898c91db64b4029ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d93deaa6b5407cafbf961bfcf8665b" is_pre_election: true
I20260201 03:02:48.146685 24895 raft_consensus.cc:2468] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a5e47ca4c894c898c91db64b4029ca6 in term 0.
I20260201 03:02:48.146940 24569 leader_election.cc:304] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a5e47ca4c894c898c91db64b4029ca6, de95131e07f04a5ab7724dadde2c4918; no voters:
I20260201 03:02:48.147222 24966 raft_consensus.cc:2804] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:02:48.147289 24966 raft_consensus.cc:493] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:02:48.147317 24966 raft_consensus.cc:3060] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:48.148298 24966 raft_consensus.cc:515] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de95131e07f04a5ab7724dadde2c4918" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 34847 } } peers { permanent_uuid: "9a5e47ca4c894c898c91db64b4029ca6" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 40941 } } peers { permanent_uuid: "41d93deaa6b5407cafbf961bfcf8665b" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33547 } }
I20260201 03:02:48.148627 24966 leader_election.cc:290] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [CANDIDATE]: Term 1 election: Requested vote from peers de95131e07f04a5ab7724dadde2c4918 (127.18.203.194:34847), 41d93deaa6b5407cafbf961bfcf8665b (127.18.203.195:33547)
I20260201 03:02:48.148972 24764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f3fed0cd0d04a6e97a5fba9d43dcf8b" candidate_uuid: "9a5e47ca4c894c898c91db64b4029ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de95131e07f04a5ab7724dadde2c4918"
I20260201 03:02:48.148981 24895 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f3fed0cd0d04a6e97a5fba9d43dcf8b" candidate_uuid: "9a5e47ca4c894c898c91db64b4029ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d93deaa6b5407cafbf961bfcf8665b"
I20260201 03:02:48.149081 24895 raft_consensus.cc:3060] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:48.149082 24764 raft_consensus.cc:3060] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:48.149991 24764 raft_consensus.cc:2468] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a5e47ca4c894c898c91db64b4029ca6 in term 1.
I20260201 03:02:48.150296 24569 leader_election.cc:304] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a5e47ca4c894c898c91db64b4029ca6, de95131e07f04a5ab7724dadde2c4918; no voters:
I20260201 03:02:48.150463 24895 raft_consensus.cc:2468] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a5e47ca4c894c898c91db64b4029ca6 in term 1.
I20260201 03:02:48.150465 24966 raft_consensus.cc:2804] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:48.150678 24966 raft_consensus.cc:697] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 9a5e47ca4c894c898c91db64b4029ca6, State: Running, Role: LEADER
I20260201 03:02:48.151005 24966 consensus_queue.cc:237] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de95131e07f04a5ab7724dadde2c4918" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 34847 } } peers { permanent_uuid: "9a5e47ca4c894c898c91db64b4029ca6" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 40941 } } peers { permanent_uuid: "41d93deaa6b5407cafbf961bfcf8665b" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33547 } }
I20260201 03:02:48.154206 24490 catalog_manager.cc:5654] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a5e47ca4c894c898c91db64b4029ca6 (127.18.203.193). New cstate: current_term: 1 leader_uuid: "9a5e47ca4c894c898c91db64b4029ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de95131e07f04a5ab7724dadde2c4918" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a5e47ca4c894c898c91db64b4029ca6" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 40941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41d93deaa6b5407cafbf961bfcf8665b" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33547 } health_report { overall_health: UNKNOWN } } }
I20260201 03:02:48.176017 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:48.174657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.18.203.193:40941)
0201 03:02:48.174781 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:48.174785 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:48.174799 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:02:48.174886 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:48.174889 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:48.174945 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:48.175014 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:48.175671 (+ 657us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:48.175674 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:48.175676 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:48.175684 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:48.175703 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:48.175709 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:48.175781 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:48.175841 (+ 60us) server_negotiation.cc:300] Negotiation successful
0201 03:02:48.175874 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260201 03:02:48.181535 24764 raft_consensus.cc:1275] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918 [term 1 FOLLOWER]: Refusing update from remote peer 9a5e47ca4c894c898c91db64b4029ca6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:48.181535 24895 raft_consensus.cc:1275] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b [term 1 FOLLOWER]: Refusing update from remote peer 9a5e47ca4c894c898c91db64b4029ca6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:02:48.182132 24966 consensus_queue.cc:1048] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41d93deaa6b5407cafbf961bfcf8665b" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:48.182315 24969 consensus_queue.cc:1048] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 9a5e47ca4c894c898c91db64b4029ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de95131e07f04a5ab7724dadde2c4918" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:02:48.196599 24490 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:48.196792 24490 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:02:48.199157 24490 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 68d87e4571744a80b24cbc771aef5fd6: Sending DeleteTablet for 3 replicas of tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b
I20260201 03:02:48.199965 24744 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:48 UTC) from {username='slave'} at 127.0.0.1:42294
I20260201 03:02:48.199983 24875 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:48 UTC) from {username='slave'} at 127.0.0.1:36772
I20260201 03:02:48.200172 24613 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:02:48 UTC) from {username='slave'} at 127.0.0.1:35066
I20260201 03:02:48.200448 24981 tablet_replica.cc:333] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918: stopping tablet replica
I20260201 03:02:48.200587 24982 tablet_replica.cc:333] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b: stopping tablet replica
I20260201 03:02:48.200732 24981 raft_consensus.cc:2243] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:48.200873 24982 raft_consensus.cc:2243] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:02:48.201012 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 24551
I20260201 03:02:48.201118 24981 raft_consensus.cc:2272] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:48.201193 24982 raft_consensus.cc:2272] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:02:48.203083 24982 ts_tablet_manager.cc:1916] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:48.203083 24981 ts_tablet_manager.cc:1916] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:02:48.205204 24981 ts_tablet_manager.cc:1929] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:02:48.205291 24981 log.cc:1199] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/2f3fed0cd0d04a6e97a5fba9d43dcf8b
I20260201 03:02:48.205574 24981 ts_tablet_manager.cc:1950] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P de95131e07f04a5ab7724dadde2c4918: Deleting consensus metadata
I20260201 03:02:48.205554 24982 ts_tablet_manager.cc:1929] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:02:48.205636 24982 log.cc:1199] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/2f3fed0cd0d04a6e97a5fba9d43dcf8b
I20260201 03:02:48.205968 24982 ts_tablet_manager.cc:1950] T 2f3fed0cd0d04a6e97a5fba9d43dcf8b P 41d93deaa6b5407cafbf961bfcf8665b: Deleting consensus metadata
I20260201 03:02:48.206715 24478 catalog_manager.cc:4985] TS de95131e07f04a5ab7724dadde2c4918 (127.18.203.194:34847): tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b (table test-table [id=b347c210cdba4d52b3f12d340bfca9cd]) successfully deleted
I20260201 03:02:48.207067 24475 catalog_manager.cc:4985] TS 41d93deaa6b5407cafbf961bfcf8665b (127.18.203.195:33547): tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b (table test-table [id=b347c210cdba4d52b3f12d340bfca9cd]) successfully deleted
W20260201 03:02:48.209403 24478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.203.193:40941 (error 108)
I20260201 03:02:48.209683 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 24682
I20260201 03:02:48.210813 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:48.210522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:40941 (local address 127.0.0.1:35084)
0201 03:02:48.210658 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:48.210725 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.193:40941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260201 03:02:48.211047 24478 catalog_manager.cc:4712] TS 9a5e47ca4c894c898c91db64b4029ca6 (127.18.203.193:40941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f3fed0cd0d04a6e97a5fba9d43dcf8b: Network error: Client connection negotiation failed: client connection to 127.18.203.193:40941: connect: Connection refused (error 111)
I20260201 03:02:48.215910 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 24813
I20260201 03:02:48.221228 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 24460
2026-02-01T03:02:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1009 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260201 03:02:48.242890 19247 mini_oidc.cc:150] Starting JWKS server
I20260201 03:02:48.244508 19247 webserver.cc:492] Webserver started at https://127.0.0.1:45501/ using document root <none> and password file <none>
I20260201 03:02:48.244652 19247 mini_oidc.cc:186] Starting OIDC Discovery server
I20260201 03:02:48.244855 19247 webserver.cc:492] Webserver started at http://127.0.0.1:35303/ using document root <none> and password file <none>
2026-02-01T03:02:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:02:48Z Disabled control of system clock
I20260201 03:02:48.253320 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:45935
--webserver_interface=127.18.203.254
--webserver_port=0
--builtin_ntp_servers=127.18.203.212:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:45935
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45501/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/testchainca.cert with env {}
W20260201 03:02:48.362279 24990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:02:48.362564 24990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:02:48.362638 24990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:02:48.366044 24990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:02:48.366134 24990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:02:48.366159 24990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:02:48.366183 24990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:02:48.366209 24990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260201 03:02:48.366242 24990 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260201 03:02:48.366266 24990 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45501/jwks.json
I20260201 03:02:48.370754 24990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:45935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:45935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45501/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:02:48.371981 24990 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:02:48.372828 24990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:02:48.378309 24995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:02:48.378317 24998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:48.378582 24990 server_base.cc:1060] running on GCE node
W20260201 03:02:48.378319 24996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:02:48.379177 24990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:02:48.380157 24990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:02:48.381304 24990 hybrid_clock.cc:648] HybridClock initialized: now 1769914968381294 us; error 29 us; skew 500 ppm
I20260201 03:02:48.383049 24990 webserver.cc:492] Webserver started at http://127.18.203.254:39449/ using document root <none> and password file <none>
I20260201 03:02:48.383638 24990 fs_manager.cc:362] Metadata directory not provided
I20260201 03:02:48.383745 24990 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:02:48.383955 24990 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:02:48.385624 24990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "c0fc627a66e44ebb9aa034c4c1dbf47e"
format_stamp: "Formatted at 2026-02-01 03:02:48 on dist-test-slave-gf59"
server_key: "de064b3d380f1ed7d7a8353bfe1e8d2a"
server_key_iv: "7ee27f71e6f5aaa754afdc87c637bc32"
server_key_version: "encryptionkey@0"
I20260201 03:02:48.386065 24990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "c0fc627a66e44ebb9aa034c4c1dbf47e"
format_stamp: "Formatted at 2026-02-01 03:02:48 on dist-test-slave-gf59"
server_key: "de064b3d380f1ed7d7a8353bfe1e8d2a"
server_key_iv: "7ee27f71e6f5aaa754afdc87c637bc32"
server_key_version: "encryptionkey@0"
I20260201 03:02:48.389303 24990 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:02:48.391389 25004 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:02:48.392423 24990 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:02:48.392532 24990 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "c0fc627a66e44ebb9aa034c4c1dbf47e"
format_stamp: "Formatted at 2026-02-01 03:02:48 on dist-test-slave-gf59"
server_key: "de064b3d380f1ed7d7a8353bfe1e8d2a"
server_key_iv: "7ee27f71e6f5aaa754afdc87c637bc32"
server_key_version: "encryptionkey@0"
I20260201 03:02:48.392621 24990 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:02:48.404431 24990 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:02:48.405121 24990 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:02:48.405303 24990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:02:48.412468 25056 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:45935 every 8 connection(s)
I20260201 03:02:48.412472 24990 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:45935
I20260201 03:02:48.413525 24990 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:02:48.416313 25057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:02:48.417775 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 24990
I20260201 03:02:48.417907 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:02:48.418184 19247 external_mini_cluster.cc:1468] Setting key f42c6117122534fdfd821f11d434a700
I20260201 03:02:48.422391 25057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e: Bootstrap starting.
I20260201 03:02:48.424742 25057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e: Neither blocks nor log segments found. Creating new log.
I20260201 03:02:48.425436 25057 log.cc:826] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e: Log is configured to *not* fsync() on all Append() calls
I20260201 03:02:48.427155 25057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e: No bootstrap required, opened a new log
I20260201 03:02:48.429276 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:48.419392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.18.203.254:45935)
0201 03:02:48.420012 (+ 620us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:48.420021 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:48.420058 (+ 37us) server_negotiation.cc:408] Connection header received
0201 03:02:48.420562 (+ 504us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:48.420579 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:48.420913 (+ 334us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:48.421197 (+ 284us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:02:48.421739 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:48.422510 (+ 771us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:48.423345 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:48.423572 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:48.427726 (+ 4154us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:02:48.427746 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:02:48.427756 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:02:48.428132 (+ 376us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:02:48.428178 (+ 46us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:02:48.428193 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:02:48.428282 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:48.428455 (+ 173us) server_negotiation.cc:300] Negotiation successful
0201 03:02:48.428691 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":107,"threads_started":1}
I20260201 03:02:48.429868 25057 raft_consensus.cc:359] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fc627a66e44ebb9aa034c4c1dbf47e" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 45935 } }
I20260201 03:02:48.430059 25057 raft_consensus.cc:385] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:02:48.430124 25057 raft_consensus.cc:740] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fc627a66e44ebb9aa034c4c1dbf47e, State: Initialized, Role: FOLLOWER
I20260201 03:02:48.430580 25057 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fc627a66e44ebb9aa034c4c1dbf47e" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 45935 } }
I20260201 03:02:48.430725 25057 raft_consensus.cc:399] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:02:48.430786 25057 raft_consensus.cc:493] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:02:48.430877 25057 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:02:48.431809 25057 raft_consensus.cc:515] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fc627a66e44ebb9aa034c4c1dbf47e" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 45935 } }
I20260201 03:02:48.431876 19247 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260201 03:02:48.432145 25057 leader_election.cc:304] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0fc627a66e44ebb9aa034c4c1dbf47e; no voters:
I20260201 03:02:48.432421 25057 leader_election.cc:290] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:02:48.432511 25062 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:02:48.432849 25062 raft_consensus.cc:697] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [term 1 LEADER]: Becoming Leader. State: Replica: c0fc627a66e44ebb9aa034c4c1dbf47e, State: Running, Role: LEADER
I20260201 03:02:48.433153 25062 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fc627a66e44ebb9aa034c4c1dbf47e" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 45935 } }
I20260201 03:02:48.433497 25057 sys_catalog.cc:565] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:02:48.434281 25062 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0fc627a66e44ebb9aa034c4c1dbf47e. Latest consensus state: current_term: 1 leader_uuid: "c0fc627a66e44ebb9aa034c4c1dbf47e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fc627a66e44ebb9aa034c4c1dbf47e" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 45935 } } }
I20260201 03:02:48.434397 25062 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:48.435201 25064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0fc627a66e44ebb9aa034c4c1dbf47e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fc627a66e44ebb9aa034c4c1dbf47e" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 45935 } } }
I20260201 03:02:48.435364 25064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e [sys.catalog]: This master's current role is: LEADER
I20260201 03:02:48.435755 25071 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:02:48.438189 25071 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:02:48.443357 25071 catalog_manager.cc:1357] Generated new cluster ID: 3778319ed94040fdb4f8b0738eb50a1e
I20260201 03:02:48.443437 25071 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:02:48.467936 25071 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:02:48.468890 25071 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:02:48.478780 25071 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c0fc627a66e44ebb9aa034c4c1dbf47e: Generated new TSK 0
I20260201 03:02:48.479552 25071 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:02:48.517875 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:48.493282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.18.203.254:45935)
0201 03:02:48.493432 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:48.493437 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:48.493554 (+ 117us) server_negotiation.cc:408] Connection header received
0201 03:02:48.493773 (+ 219us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:48.493777 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:48.493846 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:48.493994 (+ 148us) server_negotiation.cc:227] Negotiated authn=JWT
0201 03:02:48.494373 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:48.495209 (+ 836us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:48.496025 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:48.496234 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:48.496358 (+ 124us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0201 03:02:48.517352 (+ 20994us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0201 03:02:48.517484 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0201 03:02:48.517631 (+ 147us) server_negotiation.cc:300] Negotiation successful
0201 03:02:48.517702 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"thread_start_us":170,"threads_started":1}
I20260201 03:02:48.529932 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:02:48.525731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.18.203.254:45935)
0201 03:02:48.525871 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0201 03:02:48.525876 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:02:48.525974 (+ 98us) server_negotiation.cc:408] Connection header received
0201 03:02:48.526077 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:02:48.526081 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:02:48.526133 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:02:48.526214 (+ 81us) server_negotiation.cc:227] Negotiated authn=JWT
0201 03:02:48.526538 (+ 324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:48.527340 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:02:48.528084 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:02:48.528295 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:48.528350 (+ 55us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0201 03:02:48.529550 (+ 1200us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0201 03:02:48.529791 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56562: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":56}
W20260201 03:02:48.530041 25060 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56562: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260201 03:02:48.530692 25093 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:02:48.525614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:45935 (local address 127.0.0.1:56562)
0201 03:02:48.525891 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0201 03:02:48.525905 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:02:48.526001 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:02:48.526228 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:02:48.526230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:02:48.526236 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0201 03:02:48.526448 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:48.526454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:48.527491 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:02:48.527495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:02:48.527975 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:02:48.527987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:02:48.528078 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:02:48.528100 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0201 03:02:48.529759 (+ 1659us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0201 03:02:48.529915 (+ 156us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.203.254:45935: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
I20260201 03:03:03.538566 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:03.535140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.18.203.254:45935)
0201 03:03:03.535448 (+ 308us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:03.535453 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:03.535469 (+ 16us) server_negotiation.cc:408] Connection header received
0201 03:03:03.535532 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:03.535536 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:03.535587 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:03.535691 (+ 104us) server_negotiation.cc:227] Negotiated authn=JWT
0201 03:03:03.536125 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:03.536689 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:03.537373 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:03.537549 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:03.537598 (+ 49us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0201 03:03:03.538227 (+ 629us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0201 03:03:03.538412 (+ 185us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35248: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":120,"threads_started":1}
W20260201 03:03:03.538632 25098 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:03.534935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:45935 (local address 127.0.0.1:35248)
0201 03:03:03.535266 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:03.535282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:03.535410 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:03.535727 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:03.535730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:03.535739 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0201 03:03:03.535985 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:03.535995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:03.536816 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:03.536820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:03.537252 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:03.537260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:03.537366 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:03.537394 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0201 03:03:03.538354 (+ 960us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0201 03:03:03.538498 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.203.254:45935: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":155,"threads_started":1}
W20260201 03:03:03.538664 25099 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35248: JWT verification failed: token expired
W20260201 03:03:03.541373 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:03.540606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.18.203.254:45935)
0201 03:03:03.540759 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:03.540763 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:03.540812 (+ 49us) server_negotiation.cc:408] Connection header received
0201 03:03:03.540924 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:03.540928 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:03.541003 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:03.541070 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:03.541283 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35250: BlockingRecv error: recv got EOF from 127.0.0.1:35250 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260201 03:03:03.544797 25109 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.18.203.254:45935
W20260201 03:03:03.545393 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:03.544530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.18.203.254:45935)
0201 03:03:03.544666 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:03.544671 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:03.544735 (+ 64us) server_negotiation.cc:408] Connection header received
0201 03:03:03.544990 (+ 255us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:03.544993 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:03.545039 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:03.545110 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:03.545298 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35254: BlockingRecv error: recv got EOF from 127.0.0.1:35254 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260201 03:03:03.545653 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 24990
2026-02-01T03:03:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15344 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260201 03:03:03.587020 19247 mini_oidc.cc:150] Starting JWKS server
I20260201 03:03:03.588019 19247 webserver.cc:492] Webserver started at https://127.0.0.1:46527/ using document root <none> and password file <none>
I20260201 03:03:03.588115 19247 mini_oidc.cc:186] Starting OIDC Discovery server
I20260201 03:03:03.588332 19247 webserver.cc:492] Webserver started at http://127.0.0.1:40649/ using document root <none> and password file <none>
2026-02-01T03:03:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:03:03Z Disabled control of system clock
I20260201 03:03:03.597772 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:34813
--webserver_interface=127.18.203.254
--webserver_port=0
--builtin_ntp_servers=127.18.203.212:38329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:34813
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46527/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/testchainca.cert with env {}
W20260201 03:03:03.708762 25115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:03.709028 25115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:03.709079 25115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:03.712443 25115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:03.712520 25115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:03.712535 25115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:03.712554 25115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:03.712584 25115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260201 03:03:03.712615 25115 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260201 03:03:03.712643 25115 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46527/jwks.json
I20260201 03:03:03.717229 25115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:38329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:34813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:34813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46527/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:03.718546 25115 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:03.719491 25115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:03.725318 25121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:03.725318 25123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:03.725342 25120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:03.725644 25115 server_base.cc:1060] running on GCE node
I20260201 03:03:03.726210 25115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:03.727126 25115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:03.728312 25115 hybrid_clock.cc:648] HybridClock initialized: now 1769914983728298 us; error 33 us; skew 500 ppm
I20260201 03:03:03.730273 25115 webserver.cc:492] Webserver started at http://127.18.203.254:43521/ using document root <none> and password file <none>
I20260201 03:03:03.730826 25115 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:03.730890 25115 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:03.731070 25115 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:03.732833 25115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "232f149df2354043ac3f98cdf4a147ea"
format_stamp: "Formatted at 2026-02-01 03:03:03 on dist-test-slave-gf59"
server_key: "9a95bc0113838d12464aa8942d7f597f"
server_key_iv: "7a9387d83e2dead123aff6b31c7e96a9"
server_key_version: "encryptionkey@0"
I20260201 03:03:03.733345 25115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "232f149df2354043ac3f98cdf4a147ea"
format_stamp: "Formatted at 2026-02-01 03:03:03 on dist-test-slave-gf59"
server_key: "9a95bc0113838d12464aa8942d7f597f"
server_key_iv: "7a9387d83e2dead123aff6b31c7e96a9"
server_key_version: "encryptionkey@0"
I20260201 03:03:03.736747 25115 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:03:03.738998 25129 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:03.740085 25115 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:03.740209 25115 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "232f149df2354043ac3f98cdf4a147ea"
format_stamp: "Formatted at 2026-02-01 03:03:03 on dist-test-slave-gf59"
server_key: "9a95bc0113838d12464aa8942d7f597f"
server_key_iv: "7a9387d83e2dead123aff6b31c7e96a9"
server_key_version: "encryptionkey@0"
I20260201 03:03:03.740303 25115 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:03.760882 25115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:03.761605 25115 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:03.761794 25115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:03.769436 25181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:34813 every 8 connection(s)
I20260201 03:03:03.769440 25115 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:34813
I20260201 03:03:03.770561 25115 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:03:03.773455 25182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:03.774189 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 25115
I20260201 03:03:03.774292 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:03:03.774531 19247 external_mini_cluster.cc:1468] Setting key b0bf962b39a9a7386c6082be07557355
I20260201 03:03:03.780288 25182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea: Bootstrap starting.
I20260201 03:03:03.782948 25182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:03.783056 25185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:03.775819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.18.203.254:34813)
0201 03:03:03.776575 (+ 756us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:03.776586 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:03.776630 (+ 44us) server_negotiation.cc:408] Connection header received
0201 03:03:03.777307 (+ 677us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:03.777334 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:03.777779 (+ 445us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:03.778147 (+ 368us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:03.778667 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:03.779827 (+ 1160us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:03.780493 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:03.780777 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:03.781196 (+ 419us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:03.781242 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:03.781254 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:03:03.781757 (+ 503us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:03.781814 (+ 57us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:03:03.781833 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:03.781943 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:03.782047 (+ 104us) server_negotiation.cc:300] Negotiation successful
0201 03:03:03.782282 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":117,"threads_started":1}
I20260201 03:03:03.783776 25182 log.cc:826] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:03.785724 25182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea: No bootstrap required, opened a new log
I20260201 03:03:03.785866 19247 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260201 03:03:03.788408 25182 raft_consensus.cc:359] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232f149df2354043ac3f98cdf4a147ea" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34813 } }
I20260201 03:03:03.788642 25182 raft_consensus.cc:385] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:03.788705 25182 raft_consensus.cc:740] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232f149df2354043ac3f98cdf4a147ea, State: Initialized, Role: FOLLOWER
I20260201 03:03:03.789328 25182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232f149df2354043ac3f98cdf4a147ea" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34813 } }
I20260201 03:03:03.789501 25182 raft_consensus.cc:399] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:03:03.789567 25182 raft_consensus.cc:493] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:03:03.789656 25182 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:03.790563 25182 raft_consensus.cc:515] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232f149df2354043ac3f98cdf4a147ea" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34813 } }
I20260201 03:03:03.790907 25182 leader_election.cc:304] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 232f149df2354043ac3f98cdf4a147ea; no voters:
I20260201 03:03:03.791199 25182 leader_election.cc:290] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:03:03.791302 25187 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:03.791507 25187 raft_consensus.cc:697] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [term 1 LEADER]: Becoming Leader. State: Replica: 232f149df2354043ac3f98cdf4a147ea, State: Running, Role: LEADER
I20260201 03:03:03.791841 25187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232f149df2354043ac3f98cdf4a147ea" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34813 } }
I20260201 03:03:03.792456 25182 sys_catalog.cc:565] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:03.793560 25189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 232f149df2354043ac3f98cdf4a147ea. Latest consensus state: current_term: 1 leader_uuid: "232f149df2354043ac3f98cdf4a147ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232f149df2354043ac3f98cdf4a147ea" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34813 } } }
I20260201 03:03:03.793663 25188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "232f149df2354043ac3f98cdf4a147ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232f149df2354043ac3f98cdf4a147ea" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 34813 } } }
I20260201 03:03:03.793789 25188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:03.793789 25189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:03.794203 25196 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:03:03.797683 25196 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:03:03.803320 25196 catalog_manager.cc:1357] Generated new cluster ID: 5952299711e64d308f7dadc55075ec6c
I20260201 03:03:03.803388 25196 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:03:03.832176 25196 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:03:03.833302 25196 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:03:03.842582 25196 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 232f149df2354043ac3f98cdf4a147ea: Generated new TSK 0
I20260201 03:03:03.843524 25196 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260201 03:03:03.884320 25212 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260201 03:03:03.884418 25212 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260201 03:03:03.885007 25185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:03.861042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.18.203.254:34813)
0201 03:03:03.861192 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:03.861197 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:03.861332 (+ 135us) server_negotiation.cc:408] Connection header received
0201 03:03:03.861516 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:03.861520 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:03.861574 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:03.861662 (+ 88us) server_negotiation.cc:227] Negotiated authn=JWT
0201 03:03:03.862093 (+ 431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:03.862622 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:03.863578 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:03.863778 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:03.863886 (+ 108us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0201 03:03:03.884623 (+ 20737us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0201 03:03:03.884886 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":66}
W20260201 03:03:03.885085 25185 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260201 03:03:03.885228 25211 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:03.860880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:34813 (local address 127.0.0.1:50408)
0201 03:03:03.861242 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:03.861258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:03.861384 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:03.861678 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:03.861681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:03.861690 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0201 03:03:03.861993 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:03.862002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:03.862756 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:03.862760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:03.863446 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:03.863459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:03.863579 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:03.863634 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0201 03:03:03.884842 (+ 21208us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0201 03:03:03.885014 (+ 172us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.203.254:34813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":164,"threads_started":1}
I20260201 03:03:03.885914 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 25115
2026-02-01T03:03:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (339 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260201 03:03:03.926085 19247 mini_oidc.cc:150] Starting JWKS server
I20260201 03:03:03.927000 19247 webserver.cc:492] Webserver started at https://127.0.0.1:41617/ using document root <none> and password file <none>
I20260201 03:03:03.927098 19247 mini_oidc.cc:186] Starting OIDC Discovery server
I20260201 03:03:03.927330 19247 webserver.cc:492] Webserver started at http://127.0.0.1:41565/ using document root <none> and password file <none>
2026-02-01T03:03:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:03:03Z Disabled control of system clock
I20260201 03:03:03.936192 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:39217
--webserver_interface=127.18.203.254
--webserver_port=0
--builtin_ntp_servers=127.18.203.212:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:39217
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41617/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260201 03:03:04.046711 25218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:04.046989 25218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:04.047060 25218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:04.050505 25218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:04.050598 25218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:04.050619 25218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:04.050644 25218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:04.050668 25218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260201 03:03:04.050696 25218 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260201 03:03:04.050724 25218 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41617/jwks.json
I20260201 03:03:04.054854 25218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:39217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:39217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41617/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:04.056087 25218 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:04.057402 25218 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:04.063529 25224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:04.063700 25218 server_base.cc:1060] running on GCE node
W20260201 03:03:04.063529 25226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:04.063532 25223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:04.064338 25218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:04.065333 25218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:04.066496 25218 hybrid_clock.cc:648] HybridClock initialized: now 1769914984066487 us; error 40 us; skew 500 ppm
I20260201 03:03:04.068564 25218 webserver.cc:492] Webserver started at http://127.18.203.254:43247/ using document root <none> and password file <none>
I20260201 03:03:04.069155 25218 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:04.069229 25218 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:04.069433 25218 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:04.071183 25218 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "decffdf1a95c4875a7fb1d2c986daa67"
format_stamp: "Formatted at 2026-02-01 03:03:04 on dist-test-slave-gf59"
server_key: "2f88619c3b1424584e9acc302599aa31"
server_key_iv: "075bd5ed3ad03b77f7058acdedb75fd3"
server_key_version: "encryptionkey@0"
I20260201 03:03:04.071759 25218 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "decffdf1a95c4875a7fb1d2c986daa67"
format_stamp: "Formatted at 2026-02-01 03:03:04 on dist-test-slave-gf59"
server_key: "2f88619c3b1424584e9acc302599aa31"
server_key_iv: "075bd5ed3ad03b77f7058acdedb75fd3"
server_key_version: "encryptionkey@0"
I20260201 03:03:04.075246 25218 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260201 03:03:04.077733 25232 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:04.078838 25218 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:04.078959 25218 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "decffdf1a95c4875a7fb1d2c986daa67"
format_stamp: "Formatted at 2026-02-01 03:03:04 on dist-test-slave-gf59"
server_key: "2f88619c3b1424584e9acc302599aa31"
server_key_iv: "075bd5ed3ad03b77f7058acdedb75fd3"
server_key_version: "encryptionkey@0"
I20260201 03:03:04.079069 25218 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:04.095391 25218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:04.096151 25218 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:04.096346 25218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:04.103492 25218 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:39217
I20260201 03:03:04.103497 25284 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:39217 every 8 connection(s)
I20260201 03:03:04.104635 25218 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:03:04.107695 25285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:04.112102 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 25218
I20260201 03:03:04.112227 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:03:04.112442 19247 external_mini_cluster.cc:1468] Setting key 05a24bb6113e0e7264b0e61a0fb3801b
I20260201 03:03:04.114066 25285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67: Bootstrap starting.
I20260201 03:03:04.116341 25285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:04.117092 25285 log.cc:826] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:04.119683 25285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67: No bootstrap required, opened a new log
I20260201 03:03:04.120261 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:04.113684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.18.203.254:39217)
0201 03:03:04.114396 (+ 712us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:04.114406 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:04.114445 (+ 39us) server_negotiation.cc:408] Connection header received
0201 03:03:04.115029 (+ 584us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:04.115047 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:04.115416 (+ 369us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:04.115775 (+ 359us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:04.116278 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:04.117083 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:04.117777 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:04.118082 (+ 305us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:04.118572 (+ 490us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:04.118594 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:04.118610 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:03:04.119024 (+ 414us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:04.119072 (+ 48us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:03:04.119087 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:04.119185 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:04.119347 (+ 162us) server_negotiation.cc:300] Negotiation successful
0201 03:03:04.119598 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":153,"threads_started":1}
I20260201 03:03:04.122711 25285 raft_consensus.cc:359] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decffdf1a95c4875a7fb1d2c986daa67" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 39217 } }
I20260201 03:03:04.122983 25285 raft_consensus.cc:385] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:04.123049 25285 raft_consensus.cc:740] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decffdf1a95c4875a7fb1d2c986daa67, State: Initialized, Role: FOLLOWER
I20260201 03:03:04.123286 19247 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260201 03:03:04.123528 25285 consensus_queue.cc:260] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decffdf1a95c4875a7fb1d2c986daa67" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 39217 } }
I20260201 03:03:04.123693 25285 raft_consensus.cc:399] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:03:04.123759 25285 raft_consensus.cc:493] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:03:04.123855 25285 raft_consensus.cc:3060] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:04.125018 25285 raft_consensus.cc:515] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decffdf1a95c4875a7fb1d2c986daa67" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 39217 } }
I20260201 03:03:04.125433 25285 leader_election.cc:304] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: decffdf1a95c4875a7fb1d2c986daa67; no voters:
I20260201 03:03:04.125792 25285 leader_election.cc:290] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:03:04.125883 25290 raft_consensus.cc:2804] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:04.126093 25290 raft_consensus.cc:697] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [term 1 LEADER]: Becoming Leader. State: Replica: decffdf1a95c4875a7fb1d2c986daa67, State: Running, Role: LEADER
I20260201 03:03:04.126361 25290 consensus_queue.cc:237] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decffdf1a95c4875a7fb1d2c986daa67" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 39217 } }
I20260201 03:03:04.126920 25285 sys_catalog.cc:565] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:04.128067 25291 sys_catalog.cc:455] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "decffdf1a95c4875a7fb1d2c986daa67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decffdf1a95c4875a7fb1d2c986daa67" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 39217 } } }
I20260201 03:03:04.128177 25291 sys_catalog.cc:458] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:04.128521 25299 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:03:04.129195 25292 sys_catalog.cc:455] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader decffdf1a95c4875a7fb1d2c986daa67. Latest consensus state: current_term: 1 leader_uuid: "decffdf1a95c4875a7fb1d2c986daa67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decffdf1a95c4875a7fb1d2c986daa67" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 39217 } } }
I20260201 03:03:04.129357 25292 sys_catalog.cc:458] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:04.131196 25299 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:03:04.136502 25299 catalog_manager.cc:1357] Generated new cluster ID: eaeb0ecd7f154523bf98e26a46c706d0
I20260201 03:03:04.136596 25299 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:03:04.147310 25299 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:03:04.148348 25299 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:03:04.158541 25299 catalog_manager.cc:6027] T 00000000000000000000000000000000 P decffdf1a95c4875a7fb1d2c986daa67: Generated new TSK 0
I20260201 03:03:04.159310 25299 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260201 03:03:04.195348 25315 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260201 03:03:04.195451 25315 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260201 03:03:04.196866 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:04.171555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.18.203.254:39217)
0201 03:03:04.171731 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:04.171735 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:04.171849 (+ 114us) server_negotiation.cc:408] Connection header received
0201 03:03:04.172115 (+ 266us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:04.172119 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:04.172183 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:04.172284 (+ 101us) server_negotiation.cc:227] Negotiated authn=JWT
0201 03:03:04.172731 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:04.173299 (+ 568us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:04.174221 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:04.174372 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:04.174477 (+ 105us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0201 03:03:04.196428 (+ 21951us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0201 03:03:04.196742 (+ 314us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49094: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":94}
W20260201 03:03:04.196936 25288 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49094: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260201 03:03:04.197022 25314 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:04.171351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:39217 (local address 127.0.0.1:49094)
0201 03:03:04.171747 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:04.171764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:04.171905 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:04.172306 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:04.172309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:04.172319 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0201 03:03:04.172621 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:04.172631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:04.173408 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:04.173412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:04.174101 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:04.174113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:04.174245 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:04.174275 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0201 03:03:04.196642 (+ 22367us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0201 03:03:04.196816 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.203.254:39217: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":141,"threads_started":1}
I20260201 03:03:04.197623 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 25218
2026-02-01T03:03:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-02-01T03:03:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:03:04Z Disabled control of system clock
I20260201 03:03:04.247359 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:35443
--webserver_interface=127.18.203.254
--webserver_port=0
--builtin_ntp_servers=127.18.203.212:43521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:35443
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/testchainca.cert with env {}
W20260201 03:03:04.358532 25319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:04.358812 25319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:04.358865 25319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:04.362205 25319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:04.362279 25319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:04.362295 25319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:04.362313 25319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:04.362329 25319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260201 03:03:04.362351 25319 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260201 03:03:04.362370 25319 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260201 03:03:04.366886 25319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:43521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:35443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:35443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:04.368093 25319 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:04.368960 25319 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:04.374874 25325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:04.374879 25324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:04.375084 25319 server_base.cc:1060] running on GCE node
W20260201 03:03:04.374878 25327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:04.375804 25319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:04.376767 25319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:04.377918 25319 hybrid_clock.cc:648] HybridClock initialized: now 1769914984377908 us; error 41 us; skew 500 ppm
I20260201 03:03:04.379914 25319 webserver.cc:492] Webserver started at http://127.18.203.254:40813/ using document root <none> and password file <none>
I20260201 03:03:04.380517 25319 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:04.380578 25319 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:04.380772 25319 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:04.382577 25319 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "b6bca6e9d00b4d7681f83d3b1b973f76"
format_stamp: "Formatted at 2026-02-01 03:03:04 on dist-test-slave-gf59"
server_key: "d3e448703561347952d79b53b831ba46"
server_key_iv: "f851178f7b30cd160894514989ba91e3"
server_key_version: "encryptionkey@0"
I20260201 03:03:04.383090 25319 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "b6bca6e9d00b4d7681f83d3b1b973f76"
format_stamp: "Formatted at 2026-02-01 03:03:04 on dist-test-slave-gf59"
server_key: "d3e448703561347952d79b53b831ba46"
server_key_iv: "f851178f7b30cd160894514989ba91e3"
server_key_version: "encryptionkey@0"
I20260201 03:03:04.386593 25319 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260201 03:03:04.389051 25333 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:04.390189 25319 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:04.390311 25319 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "b6bca6e9d00b4d7681f83d3b1b973f76"
format_stamp: "Formatted at 2026-02-01 03:03:04 on dist-test-slave-gf59"
server_key: "d3e448703561347952d79b53b831ba46"
server_key_iv: "f851178f7b30cd160894514989ba91e3"
server_key_version: "encryptionkey@0"
I20260201 03:03:04.390419 25319 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:04.413307 25319 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:04.414047 25319 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:04.414240 25319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:04.421761 25319 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:35443
I20260201 03:03:04.421757 25385 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:35443 every 8 connection(s)
I20260201 03:03:04.422886 25319 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:03:04.423202 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 25319
I20260201 03:03:04.423316 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:03:04.423593 19247 external_mini_cluster.cc:1468] Setting key f9ce625a1f4b1e5378fdb179921b906c
I20260201 03:03:04.426265 25386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:04.432004 25386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76: Bootstrap starting.
I20260201 03:03:04.432065 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:04.424897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44472 (local address 127.18.203.254:35443)
0201 03:03:04.425525 (+ 628us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:04.425534 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:04.425574 (+ 40us) server_negotiation.cc:408] Connection header received
0201 03:03:04.426238 (+ 664us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:04.426257 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:04.426648 (+ 391us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:04.426946 (+ 298us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:04.427486 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:04.428712 (+ 1226us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:04.429583 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:04.429806 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:04.430379 (+ 573us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:04.430428 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:04.430439 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:03:04.430834 (+ 395us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:04.430878 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:03:04.430891 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:04.430980 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:04.431140 (+ 160us) server_negotiation.cc:300] Negotiation successful
0201 03:03:04.431388 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260201 03:03:04.434440 25386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:04.434805 19247 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260201 03:03:04.435233 25386 log.cc:826] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:04.437244 25386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76: No bootstrap required, opened a new log
I20260201 03:03:04.439831 25386 raft_consensus.cc:359] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6bca6e9d00b4d7681f83d3b1b973f76" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35443 } }
I20260201 03:03:04.440056 25386 raft_consensus.cc:385] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:04.440130 25386 raft_consensus.cc:740] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6bca6e9d00b4d7681f83d3b1b973f76, State: Initialized, Role: FOLLOWER
I20260201 03:03:04.440572 25386 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6bca6e9d00b4d7681f83d3b1b973f76" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35443 } }
I20260201 03:03:04.440698 25386 raft_consensus.cc:399] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:03:04.440758 25386 raft_consensus.cc:493] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:03:04.440850 25386 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:04.441768 25386 raft_consensus.cc:515] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6bca6e9d00b4d7681f83d3b1b973f76" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35443 } }
I20260201 03:03:04.442162 25386 leader_election.cc:304] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6bca6e9d00b4d7681f83d3b1b973f76; no voters:
I20260201 03:03:04.442493 25386 leader_election.cc:290] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:03:04.442617 25391 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:04.442827 25391 raft_consensus.cc:697] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [term 1 LEADER]: Becoming Leader. State: Replica: b6bca6e9d00b4d7681f83d3b1b973f76, State: Running, Role: LEADER
I20260201 03:03:04.443156 25391 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6bca6e9d00b4d7681f83d3b1b973f76" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35443 } }
I20260201 03:03:04.443713 25386 sys_catalog.cc:565] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:04.444792 25393 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6bca6e9d00b4d7681f83d3b1b973f76. Latest consensus state: current_term: 1 leader_uuid: "b6bca6e9d00b4d7681f83d3b1b973f76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6bca6e9d00b4d7681f83d3b1b973f76" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35443 } } }
I20260201 03:03:04.444924 25393 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:04.445338 25400 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:03:04.445470 25392 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6bca6e9d00b4d7681f83d3b1b973f76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6bca6e9d00b4d7681f83d3b1b973f76" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35443 } } }
I20260201 03:03:04.445557 25392 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:04.448427 25400 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:03:04.454479 25400 catalog_manager.cc:1357] Generated new cluster ID: f9c255895b8a4ced8cbd0e4a78352bf3
I20260201 03:03:04.454583 25400 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:03:04.473244 25400 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:03:04.474258 25400 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:03:04.482765 25400 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b6bca6e9d00b4d7681f83d3b1b973f76: Generated new TSK 0
I20260201 03:03:04.483562 25400 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:03:04.528528 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:04.496000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.18.203.254:35443)
0201 03:03:04.496144 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:04.496148 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:04.496291 (+ 143us) server_negotiation.cc:408] Connection header received
0201 03:03:04.496463 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:04.496466 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:04.496521 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:04.496607 (+ 86us) server_negotiation.cc:227] Negotiated authn=JWT
0201 03:03:04.497031 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:04.497559 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:04.498469 (+ 910us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:04.498615 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:04.498706 (+ 91us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0201 03:03:04.528029 (+ 29323us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0201 03:03:04.528359 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44476: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260201 03:03:04.528626 25389 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44476: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20260201 03:03:04.528707 25415 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:04.495871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35443 (local address 127.0.0.1:44476)
0201 03:03:04.496194 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:04.496217 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:04.496339 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:04.496626 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:04.496630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:04.496638 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0201 03:03:04.496929 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:04.496939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:04.497679 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:04.497683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:04.498367 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:04.498378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:04.498507 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:04.498536 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0201 03:03:04.528298 (+ 29762us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0201 03:03:04.528506 (+ 208us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.203.254:35443: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":119,"threads_started":1}
I20260201 03:03:04.529572 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 25319
2026-02-01T03:03:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (311 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (127 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:03:04 dist-test-slave-gf59 krb5kdc[25427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:03:04 dist-test-slave-gf59 krb5kdc[25427](info): set up 2 sockets
Feb 01 03:03:04 dist-test-slave-gf59 krb5kdc[25427](info): commencing operation
krb5kdc: starting...
W20260201 03:03:06.604351 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.800s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:03:06 dist-test-slave-gf59 krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:03:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:03:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:06.750084 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:42483
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:42483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:06.861327 25443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:06.861590 25443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:06.861670 25443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:06.865068 25443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:06.865144 25443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:06.865176 25443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:06.865201 25443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:06.865231 25443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:03:06.869673 25443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:42483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:42483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:06.870867 25443 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:06.871810 25443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:06.877447 25451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:06.877431 25449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:06.877434 25448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:06.877723 25443 server_base.cc:1060] running on GCE node
I20260201 03:03:06.878263 25443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:06.879222 25443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:06.880380 25443 hybrid_clock.cc:648] HybridClock initialized: now 1769914986880371 us; error 28 us; skew 500 ppm
Feb 01 03:03:06 dist-test-slave-gf59 krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914986, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:06.883006 25443 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:03:06.884199 25443 webserver.cc:492] Webserver started at http://127.18.203.254:37495/ using document root <none> and password file <none>
I20260201 03:03:06.884765 25443 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:06.884821 25443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:06.885005 25443 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:06.886814 25443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "b286f172c6334b7589bfd4430d01980a"
format_stamp: "Formatted at 2026-02-01 03:03:06 on dist-test-slave-gf59"
server_key: "d143eac0938218ec9b4212050dfbb3ac"
server_key_iv: "bde44e13731c12b1877a123a4b678224"
server_key_version: "encryptionkey@0"
I20260201 03:03:06.887328 25443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "b286f172c6334b7589bfd4430d01980a"
format_stamp: "Formatted at 2026-02-01 03:03:06 on dist-test-slave-gf59"
server_key: "d143eac0938218ec9b4212050dfbb3ac"
server_key_iv: "bde44e13731c12b1877a123a4b678224"
server_key_version: "encryptionkey@0"
I20260201 03:03:06.890826 25443 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260201 03:03:06.893167 25458 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:06.894291 25443 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:06.894410 25443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "b286f172c6334b7589bfd4430d01980a"
format_stamp: "Formatted at 2026-02-01 03:03:06 on dist-test-slave-gf59"
server_key: "d143eac0938218ec9b4212050dfbb3ac"
server_key_iv: "bde44e13731c12b1877a123a4b678224"
server_key_version: "encryptionkey@0"
I20260201 03:03:06.894508 25443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:06.911418 25443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:06.914197 25443 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:06.914395 25443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:06.921662 25510 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:42483 every 8 connection(s)
I20260201 03:03:06.921674 25443 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:42483
I20260201 03:03:06.922853 25443 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:03:06.925802 25511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:06.926551 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 25443
I20260201 03:03:06.926684 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:03:06.926964 19247 external_mini_cluster.cc:1468] Setting key fb69c0eab9a832c6b168382f27d19986
I20260201 03:03:06.932755 25511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a: Bootstrap starting.
Feb 01 03:03:06 dist-test-slave-gf59 krb5kdc[25427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:03:06.935663 25511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:06.936532 25511 log.cc:826] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:06.938819 25511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a: No bootstrap required, opened a new log
I20260201 03:03:06.939728 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:06.928261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.18.203.254:42483)
0201 03:03:06.928666 (+ 405us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:06.928675 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:06.928711 (+ 36us) server_negotiation.cc:408] Connection header received
0201 03:03:06.929296 (+ 585us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:06.929315 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:06.929620 (+ 305us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:06.929920 (+ 300us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:06.930993 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:06.931835 (+ 842us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:06.932696 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:06.932938 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:06.935357 (+ 2419us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:06.935375 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:06.935387 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:06.935423 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:06.937260 (+ 1837us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:06.937711 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:06.937715 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:06.937719 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:06.937793 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:06.938011 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:06.938013 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:06.938014 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:06.938350 (+ 336us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:06.938509 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:06.938824 (+ 315us) server_negotiation.cc:300] Negotiation successful
0201 03:03:06.939077 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":122,"threads_started":1}
I20260201 03:03:06.941525 25511 raft_consensus.cc:359] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b286f172c6334b7589bfd4430d01980a" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 42483 } }
I20260201 03:03:06.941774 25511 raft_consensus.cc:385] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:06.941838 25511 raft_consensus.cc:740] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b286f172c6334b7589bfd4430d01980a, State: Initialized, Role: FOLLOWER
I20260201 03:03:06.942226 25511 consensus_queue.cc:260] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b286f172c6334b7589bfd4430d01980a" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 42483 } }
I20260201 03:03:06.942342 25511 raft_consensus.cc:399] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:03:06.942386 25511 raft_consensus.cc:493] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:03:06.942452 25511 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:06.943347 25511 raft_consensus.cc:515] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b286f172c6334b7589bfd4430d01980a" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 42483 } }
I20260201 03:03:06.943714 25511 leader_election.cc:304] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b286f172c6334b7589bfd4430d01980a; no voters:
I20260201 03:03:06.944028 25511 leader_election.cc:290] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:03:06.944124 25516 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:06.944326 25516 raft_consensus.cc:697] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [term 1 LEADER]: Becoming Leader. State: Replica: b286f172c6334b7589bfd4430d01980a, State: Running, Role: LEADER
I20260201 03:03:06.944690 25516 consensus_queue.cc:237] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b286f172c6334b7589bfd4430d01980a" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 42483 } }
I20260201 03:03:06.945207 25511 sys_catalog.cc:565] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:06.946385 25517 sys_catalog.cc:455] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b286f172c6334b7589bfd4430d01980a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b286f172c6334b7589bfd4430d01980a" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 42483 } } }
I20260201 03:03:06.946508 25517 sys_catalog.cc:458] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:06.946805 25518 sys_catalog.cc:455] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b286f172c6334b7589bfd4430d01980a. Latest consensus state: current_term: 1 leader_uuid: "b286f172c6334b7589bfd4430d01980a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b286f172c6334b7589bfd4430d01980a" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 42483 } } }
I20260201 03:03:06.946947 25518 sys_catalog.cc:458] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:06.946864 25525 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:03:06.950068 25525 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:03:06.956015 25525 catalog_manager.cc:1357] Generated new cluster ID: cb639564fef242509b05967c030aedf8
I20260201 03:03:06.956216 25525 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:03:06.975057 25525 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:03:06.976536 25525 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:03:06.984706 25525 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b286f172c6334b7589bfd4430d01980a: Generated new TSK 0
I20260201 03:03:06.985677 25525 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:07.038776 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:42483
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:07.153206 25539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:07.153465 25539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:07.153522 25539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:07.156883 25539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:07.156956 25539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:07.157037 25539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:03:07.161412 25539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:42483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:07.162525 25539 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:07.163385 25539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:07.170185 25547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:07.170185 25544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:07.170188 25545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:07.170470 25539 server_base.cc:1060] running on GCE node
I20260201 03:03:07.170887 25539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:07.171473 25539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:07.172639 25539 hybrid_clock.cc:648] HybridClock initialized: now 1769914987172633 us; error 49 us; skew 500 ppm
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:07.175316 25539 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:03:07.176530 25539 webserver.cc:492] Webserver started at http://127.18.203.193:39733/ using document root <none> and password file <none>
I20260201 03:03:07.177079 25539 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:07.177129 25539 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:07.177289 25539 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:07.179096 25539 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "48210eff49f143b1867e62c3a3ab7e83"
format_stamp: "Formatted at 2026-02-01 03:03:07 on dist-test-slave-gf59"
server_key: "31f0daef3fe95334b3d725487517339d"
server_key_iv: "33e7c1fce24435d633470f3ad70aa811"
server_key_version: "encryptionkey@0"
I20260201 03:03:07.179564 25539 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "48210eff49f143b1867e62c3a3ab7e83"
format_stamp: "Formatted at 2026-02-01 03:03:07 on dist-test-slave-gf59"
server_key: "31f0daef3fe95334b3d725487517339d"
server_key_iv: "33e7c1fce24435d633470f3ad70aa811"
server_key_version: "encryptionkey@0"
I20260201 03:03:07.183220 25539 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260201 03:03:07.185592 25554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:07.186628 25539 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:07.186733 25539 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "48210eff49f143b1867e62c3a3ab7e83"
format_stamp: "Formatted at 2026-02-01 03:03:07 on dist-test-slave-gf59"
server_key: "31f0daef3fe95334b3d725487517339d"
server_key_iv: "33e7c1fce24435d633470f3ad70aa811"
server_key_version: "encryptionkey@0"
I20260201 03:03:07.186820 25539 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:07.204391 25539 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:07.206960 25539 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:07.207141 25539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:07.207695 25539 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:07.208561 25539 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:07.208609 25539 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:07.208647 25539 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:07.208667 25539 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:07.217211 25539 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:42325
I20260201 03:03:07.217240 25667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:42325 every 8 connection(s)
I20260201 03:03:07.218148 25539 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:03:07.225862 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 25539
I20260201 03:03:07.225981 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:03:07.226212 19247 external_mini_cluster.cc:1468] Setting key 1bdaf0c515c3791e99fd0f625f3d19b7
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:03:07.230338 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:07.219830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:46419 (local address 127.18.203.254:42483)
0201 03:03:07.219972 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:07.219976 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:07.220672 (+ 696us) server_negotiation.cc:408] Connection header received
0201 03:03:07.221522 (+ 850us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:07.221526 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:07.221587 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:07.221661 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:07.223335 (+ 1674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.223880 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:07.224530 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.224715 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:07.226975 (+ 2260us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:07.226993 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:07.226995 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:07.227021 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:07.228524 (+ 1503us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:07.228975 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:07.228978 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:07.228980 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:07.229033 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:07.229284 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:07.229288 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:07.229290 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:07.229439 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:07.229524 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:07.230033 (+ 509us) server_negotiation.cc:300] Negotiation successful
0201 03:03:07.230170 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260201 03:03:07.231056 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:07.220127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:42483 (local address 127.18.203.193:46419)
0201 03:03:07.220540 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:07.220575 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:07.221325 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:07.221826 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:07.221835 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:07.222221 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:07.223175 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:07.223189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.224003 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:07.224006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:07.224415 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:07.224422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.224638 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:07.225221 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:07.225261 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:07.226834 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:07.228635 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:07.228641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:07.228654 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:07.228884 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:07.229112 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:07.229115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:07.229116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:07.229203 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:07.229538 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:07.229546 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:07.229786 (+ 240us) client_negotiation.cc:769] Sending connection context
0201 03:03:07.229976 (+ 190us) client_negotiation.cc:240] Negotiation successful
0201 03:03:07.230247 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260201 03:03:07.232250 25668 heartbeater.cc:344] Connected to a master server at 127.18.203.254:42483
I20260201 03:03:07.232547 25668 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:07.233191 25668 heartbeater.cc:507] Master 127.18.203.254:42483 requested a full tablet report, sending...
I20260201 03:03:07.234756 25475 ts_manager.cc:194] Registered new tserver with Master: 48210eff49f143b1867e62c3a3ab7e83 (127.18.203.193:42325)
I20260201 03:03:07.236017 25475 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:46419
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:07.274312 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:42483
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:07.381987 25675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:07.382233 25675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:07.382319 25675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:07.385563 25675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:07.385636 25675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:07.385737 25675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:03:07.389953 25675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:42483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:07.391093 25675 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:07.391969 25675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:07.398276 25681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:07.398320 25683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:07.398296 25680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:07.398495 25675 server_base.cc:1060] running on GCE node
I20260201 03:03:07.398937 25675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:07.399534 25675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:07.400689 25675 hybrid_clock.cc:648] HybridClock initialized: now 1769914987400680 us; error 31 us; skew 500 ppm
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:07.403240 25675 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:03:07.404376 25675 webserver.cc:492] Webserver started at http://127.18.203.194:35475/ using document root <none> and password file <none>
I20260201 03:03:07.404918 25675 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:07.404970 25675 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:07.405158 25675 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:07.406869 25675 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "196e6300b4264256a0e55d39a196acd4"
format_stamp: "Formatted at 2026-02-01 03:03:07 on dist-test-slave-gf59"
server_key: "a5014663dc895b7d3b45f35cfbe74887"
server_key_iv: "7cd1b71c9774ac18f3c2726525926836"
server_key_version: "encryptionkey@0"
I20260201 03:03:07.407341 25675 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "196e6300b4264256a0e55d39a196acd4"
format_stamp: "Formatted at 2026-02-01 03:03:07 on dist-test-slave-gf59"
server_key: "a5014663dc895b7d3b45f35cfbe74887"
server_key_iv: "7cd1b71c9774ac18f3c2726525926836"
server_key_version: "encryptionkey@0"
I20260201 03:03:07.410669 25675 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:03:07.412832 25690 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:07.413817 25675 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:07.413920 25675 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "196e6300b4264256a0e55d39a196acd4"
format_stamp: "Formatted at 2026-02-01 03:03:07 on dist-test-slave-gf59"
server_key: "a5014663dc895b7d3b45f35cfbe74887"
server_key_iv: "7cd1b71c9774ac18f3c2726525926836"
server_key_version: "encryptionkey@0"
I20260201 03:03:07.414027 25675 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:07.451252 25675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:07.456595 25675 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:07.456787 25675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:07.457374 25675 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:07.458281 25675 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:07.458331 25675 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:07.458384 25675 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:07.458410 25675 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:07.467100 25675 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:33971
I20260201 03:03:07.467137 25803 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:33971 every 8 connection(s)
I20260201 03:03:07.468087 25675 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:03:07.470813 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 25675
I20260201 03:03:07.470923 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:03:07.471148 19247 external_mini_cluster.cc:1468] Setting key 8f2b6c49f6a37157116fd976d1cd62ad
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:03:07.480281 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:07.469750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:58463 (local address 127.18.203.254:42483)
0201 03:03:07.469883 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:07.469887 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:07.470554 (+ 667us) server_negotiation.cc:408] Connection header received
0201 03:03:07.471463 (+ 909us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:07.471467 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:07.471524 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:07.471594 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:07.473229 (+ 1635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.473734 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:07.474364 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.474547 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:07.476968 (+ 2421us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:07.476985 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:07.476988 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:07.477014 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:07.478351 (+ 1337us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:07.478885 (+ 534us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:07.478887 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:07.478889 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:07.478926 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:07.479198 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:07.479200 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:07.479201 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:07.479348 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:07.479414 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:07.480038 (+ 624us) server_negotiation.cc:300] Negotiation successful
0201 03:03:07.480144 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
I20260201 03:03:07.481178 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:07.470026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:42483 (local address 127.18.203.194:58463)
0201 03:03:07.470421 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:07.470457 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:07.471256 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:07.471784 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:07.471792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:07.472176 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:07.473090 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:07.473104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.473845 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:07.473848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:07.474257 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:07.474263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.474471 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:07.475066 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:07.475091 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:07.476821 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:07.478465 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:07.478471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:07.478483 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:07.478799 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:07.478998 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:07.479001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:07.479004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:07.479120 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:07.479438 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:07.479445 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:07.479780 (+ 335us) client_negotiation.cc:769] Sending connection context
0201 03:03:07.480023 (+ 243us) client_negotiation.cc:240] Negotiation successful
0201 03:03:07.480321 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":109,"threads_started":1}
I20260201 03:03:07.482214 25804 heartbeater.cc:344] Connected to a master server at 127.18.203.254:42483
I20260201 03:03:07.482437 25804 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:07.483072 25804 heartbeater.cc:507] Master 127.18.203.254:42483 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
I20260201 03:03:07.484165 25475 ts_manager.cc:194] Registered new tserver with Master: 196e6300b4264256a0e55d39a196acd4 (127.18.203.194:33971)
I20260201 03:03:07.484869 25475 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:58463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:07.519501 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:42483
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:07.626156 25811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:07.626425 25811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:07.626506 25811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:07.629796 25811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:07.629868 25811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:07.629973 25811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:03:07.634084 25811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:42483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:07.635128 25811 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:07.635988 25811 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:07.642416 25816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:07.642418 25817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:07.642449 25819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:07.642729 25811 server_base.cc:1060] running on GCE node
I20260201 03:03:07.643076 25811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:07.643642 25811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:07.644798 25811 hybrid_clock.cc:648] HybridClock initialized: now 1769914987644786 us; error 27 us; skew 500 ppm
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:07.647780 25811 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:03:07.648850 25811 webserver.cc:492] Webserver started at http://127.18.203.195:38149/ using document root <none> and password file <none>
I20260201 03:03:07.649397 25811 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:07.649448 25811 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:07.649634 25811 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:07.651388 25811 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "1cdea2b8217546b1bf0f333886fd2b2d"
format_stamp: "Formatted at 2026-02-01 03:03:07 on dist-test-slave-gf59"
server_key: "3494e6fd332961002e6c34ef75d2ae91"
server_key_iv: "ba2c26a3fc7ca4ecc90dc73bdae51629"
server_key_version: "encryptionkey@0"
I20260201 03:03:07.651889 25811 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "1cdea2b8217546b1bf0f333886fd2b2d"
format_stamp: "Formatted at 2026-02-01 03:03:07 on dist-test-slave-gf59"
server_key: "3494e6fd332961002e6c34ef75d2ae91"
server_key_iv: "ba2c26a3fc7ca4ecc90dc73bdae51629"
server_key_version: "encryptionkey@0"
I20260201 03:03:07.655230 25811 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260201 03:03:07.657425 25826 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:07.658457 25811 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:07.658565 25811 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "1cdea2b8217546b1bf0f333886fd2b2d"
format_stamp: "Formatted at 2026-02-01 03:03:07 on dist-test-slave-gf59"
server_key: "3494e6fd332961002e6c34ef75d2ae91"
server_key_iv: "ba2c26a3fc7ca4ecc90dc73bdae51629"
server_key_version: "encryptionkey@0"
I20260201 03:03:07.658659 25811 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:07.680904 25811 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:07.683729 25811 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:07.683916 25811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:07.684495 25811 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:07.685489 25811 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:07.685542 25811 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:07.685596 25811 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:07.685621 25811 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:07.694411 25811 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:38977
I20260201 03:03:07.694434 25939 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:38977 every 8 connection(s)
I20260201 03:03:07.695430 25811 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:03:07.705402 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 25811
I20260201 03:03:07.705513 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:03:07.705742 19247 external_mini_cluster.cc:1468] Setting key 1ebeccd719034b2a04461ec55ff884bb
I20260201 03:03:07.707782 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:07.697185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:34683 (local address 127.18.203.254:42483)
0201 03:03:07.697310 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:07.697313 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:07.697990 (+ 677us) server_negotiation.cc:408] Connection header received
0201 03:03:07.698868 (+ 878us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:07.698871 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:07.698932 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:07.699010 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:07.700763 (+ 1753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.701278 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:07.701913 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.702104 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:07.704390 (+ 2286us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:07.704411 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:07.704414 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:07.704441 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:07.705790 (+ 1349us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:07.706290 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:07.706294 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:07.706295 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:07.706352 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:07.706671 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:07.706674 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:07.706676 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:07.706841 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:07.706929 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:07.707447 (+ 518us) server_negotiation.cc:300] Negotiation successful
0201 03:03:07.707575 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260201 03:03:07.708534 25942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:07.697436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:42483 (local address 127.18.203.195:34683)
0201 03:03:07.697852 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:07.697888 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:07.698644 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:07.699186 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:07.699194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:07.699652 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:07.700606 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:07.700621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.701399 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:07.701401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:07.701800 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:07.701807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.702024 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:07.702623 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:07.702651 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:07.704220 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:07.705902 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:07.705908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:07.705922 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:07.706180 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:07.706442 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:07.706444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:07.706446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:07.706580 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:07.706938 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:07.706945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:07.707201 (+ 256us) client_negotiation.cc:769] Sending connection context
0201 03:03:07.707402 (+ 201us) client_negotiation.cc:240] Negotiation successful
0201 03:03:07.707715 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260201 03:03:07.709627 25940 heartbeater.cc:344] Connected to a master server at 127.18.203.254:42483
I20260201 03:03:07.709877 25940 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:07.710426 25940 heartbeater.cc:507] Master 127.18.203.254:42483 requested a full tablet report, sending...
I20260201 03:03:07.711378 25475 ts_manager.cc:194] Registered new tserver with Master: 1cdea2b8217546b1bf0f333886fd2b2d (127.18.203.195:38977)
I20260201 03:03:07.711982 25475 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:34683
I20260201 03:03:07.719657 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/test-admin.keytab.
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:07.735707 19247 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260201 03:03:07.738348 25947 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:07.737051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.18.203.193:42325)
0201 03:03:07.737364 (+ 313us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:07.737370 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:07.737390 (+ 20us) server_negotiation.cc:408] Connection header received
0201 03:03:07.737440 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:07.737446 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:07.737610 (+ 164us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:07.737736 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:07.738196 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52436: BlockingRecv error: recv got EOF from 127.0.0.1:52436 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":108,"threads_started":1}
W20260201 03:03:07.738441 19247 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/krb5cc' of type FILE
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:07.739452 19247 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:07.740377 19247 init.cc:295] Successfully reacquired a new kerberos TGT
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.193@KRBTEST.COM
I20260201 03:03:07.749033 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:07.741285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.18.203.193:42325)
0201 03:03:07.741405 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:07.741409 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:07.741443 (+ 34us) server_negotiation.cc:408] Connection header received
0201 03:03:07.741568 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:07.741570 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:07.741612 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:07.741672 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:07.742259 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.742971 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:07.743558 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.743779 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:07.745646 (+ 1867us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:07.745684 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:07.745696 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:07.745720 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:07.747413 (+ 1693us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:07.747870 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:07.747876 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:07.747877 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:07.747936 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:07.748154 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:07.748158 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:07.748160 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:07.748365 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:07.748568 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:07.748730 (+ 162us) server_negotiation.cc:300] Negotiation successful
0201 03:03:07.748878 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260201 03:03:07.752475 25947 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:07.751663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.18.203.193:42325)
0201 03:03:07.751792 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:07.751796 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:07.751845 (+ 49us) server_negotiation.cc:408] Connection header received
0201 03:03:07.751948 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:07.751950 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:07.751991 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:07.752058 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:07.752373 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52446: BlockingRecv error: recv got EOF from 127.0.0.1:52446 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260201 03:03:07.752699 19247 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/krb5cc)
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:07.753710 19247 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:07.754495 19247 init.cc:295] Successfully reacquired a new kerberos TGT
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.193@KRBTEST.COM
I20260201 03:03:07.761555 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:07.755261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.18.203.193:42325)
0201 03:03:07.755385 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:07.755389 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:07.755432 (+ 43us) server_negotiation.cc:408] Connection header received
0201 03:03:07.755520 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:07.755522 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:07.755558 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:07.755661 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:07.756187 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.756672 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:07.757208 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.757349 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:07.759159 (+ 1810us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:07.759178 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:07.759180 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:07.759204 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:07.760398 (+ 1194us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:07.760721 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:07.760724 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:07.760725 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:07.760758 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:07.760907 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:07.760909 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:07.760911 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:07.761045 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:07.761141 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:07.761293 (+ 152us) server_negotiation.cc:300] Negotiation successful
0201 03:03:07.761393 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260201 03:03:07.764088 25947 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:07.763142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.18.203.193:42325)
0201 03:03:07.763268 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:07.763272 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:07.763345 (+ 73us) server_negotiation.cc:408] Connection header received
0201 03:03:07.763483 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:07.763487 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:07.763542 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:07.763622 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:07.763992 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52470: BlockingRecv error: recv got EOF from 127.0.0.1:52470 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260201 03:03:07.764183 19247 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769914928154323-19247-0/krb5kdc/krb5cc' of type FILE
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:07.765108 19247 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:07.765878 19247 init.cc:295] Successfully reacquired a new kerberos TGT
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.193@KRBTEST.COM
I20260201 03:03:07.773020 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:07.766681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.18.203.193:42325)
0201 03:03:07.766804 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:07.766807 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:07.766865 (+ 58us) server_negotiation.cc:408] Connection header received
0201 03:03:07.766950 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:07.766953 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:07.766989 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:07.767052 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:07.767578 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.768079 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:07.768641 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:07.768760 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:07.770610 (+ 1850us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:07.770627 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:07.770629 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:07.770651 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:07.771817 (+ 1166us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:07.772139 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:07.772142 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:07.772143 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:07.772177 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:07.772323 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:07.772324 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:07.772326 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:07.772449 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:07.772536 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:07.772759 (+ 223us) server_negotiation.cc:300] Negotiation successful
0201 03:03:07.772845 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260201 03:03:07.773993 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 25539
I20260201 03:03:07.780226 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 25675
I20260201 03:03:07.786078 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 25811
I20260201 03:03:07.791548 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 25443
2026-02-01T03:03:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3009 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25960](info): set up 2 sockets
Feb 01 03:03:07 dist-test-slave-gf59 krb5kdc[25960](info): commencing operation
krb5kdc: starting...
W20260201 03:03:09.620193 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.806s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:03:09 dist-test-slave-gf59 krb5kdc[25960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:03:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:03:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
I20260201 03:03:09.760195 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:40003
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:40003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:09.868247 25976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:09.868508 25976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:09.868584 25976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:09.871896 25976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:09.871981 25976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:09.872014 25976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:09.872040 25976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:09.872071 25976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:03:09.876418 25976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:40003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:40003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:09.877552 25976 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:09.878444 25976 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:09.884048 25981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:09.884073 25982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:09.884048 25984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:09.884135 25976 server_base.cc:1060] running on GCE node
I20260201 03:03:09.884691 25976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:09.885615 25976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:09.886765 25976 hybrid_clock.cc:648] HybridClock initialized: now 1769914989886735 us; error 57 us; skew 500 ppm
Feb 01 03:03:09 dist-test-slave-gf59 krb5kdc[25960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914989, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:09.889451 25976 init.cc:377] Logged in from keytab as oryx/127.18.203.254@KRBTEST.COM (short username oryx)
I20260201 03:03:09.890717 25976 webserver.cc:492] Webserver started at http://127.18.203.254:36657/ using document root <none> and password file <none>
I20260201 03:03:09.891252 25976 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:09.891305 25976 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:09.891464 25976 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:09.893203 25976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "4cc37f0a8a064fa388608e88d86194c2"
format_stamp: "Formatted at 2026-02-01 03:03:09 on dist-test-slave-gf59"
server_key: "50ac6309bb5376435fdd303077dea015"
server_key_iv: "32d0e51ac35195420ff125672ee039df"
server_key_version: "encryptionkey@0"
I20260201 03:03:09.893641 25976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "4cc37f0a8a064fa388608e88d86194c2"
format_stamp: "Formatted at 2026-02-01 03:03:09 on dist-test-slave-gf59"
server_key: "50ac6309bb5376435fdd303077dea015"
server_key_iv: "32d0e51ac35195420ff125672ee039df"
server_key_version: "encryptionkey@0"
I20260201 03:03:09.897001 25976 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:03:09.899221 25991 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:09.900275 25976 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:09.900384 25976 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "4cc37f0a8a064fa388608e88d86194c2"
format_stamp: "Formatted at 2026-02-01 03:03:09 on dist-test-slave-gf59"
server_key: "50ac6309bb5376435fdd303077dea015"
server_key_iv: "32d0e51ac35195420ff125672ee039df"
server_key_version: "encryptionkey@0"
I20260201 03:03:09.900485 25976 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:09.940315 25976 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:09.943064 25976 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:09.943239 25976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:09.950309 25976 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:40003
I20260201 03:03:09.950309 26043 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:40003 every 8 connection(s)
I20260201 03:03:09.951380 25976 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:03:09.954165 26044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:09.957412 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 25976
I20260201 03:03:09.957561 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:03:09.957818 19247 external_mini_cluster.cc:1468] Setting key 7a86492391795c6975f71a1a5df48a3f
I20260201 03:03:09.960322 26044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2: Bootstrap starting.
I20260201 03:03:09.962419 26044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:09.963127 26044 log.cc:826] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:09.965154 26044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2: No bootstrap required, opened a new log
Feb 01 03:03:09 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:09.968328 26044 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } }
I20260201 03:03:09.968564 26044 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:09.968631 26044 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc37f0a8a064fa388608e88d86194c2, State: Initialized, Role: FOLLOWER
I20260201 03:03:09.969128 26044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } }
I20260201 03:03:09.969288 26044 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:03:09.969389 26044 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:03:09.969487 26044 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:09.970446 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:09.959167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.18.203.254:40003)
0201 03:03:09.959593 (+ 426us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:09.959603 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:09.959671 (+ 68us) server_negotiation.cc:408] Connection header received
0201 03:03:09.960210 (+ 539us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:09.960227 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:09.960546 (+ 319us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:09.960862 (+ 316us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:09.961803 (+ 941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:09.962623 (+ 820us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:09.963311 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:09.963582 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:09.965825 (+ 2243us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:09.965845 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:09.965859 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:09.965901 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:09.967854 (+ 1953us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:09.968231 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:09.968237 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:09.968242 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:09.968353 (+ 111us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:09.968541 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:09.968544 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:09.968546 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:09.968991 (+ 445us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:09.969204 (+ 213us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:09.969412 (+ 208us) server_negotiation.cc:300] Negotiation successful
0201 03:03:09.969677 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":140,"threads_started":1}
I20260201 03:03:09.970544 26044 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } }
I20260201 03:03:09.970893 26044 leader_election.cc:304] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cc37f0a8a064fa388608e88d86194c2; no voters:
I20260201 03:03:09.971184 26044 leader_election.cc:290] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:03:09.971289 26049 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:09.971485 26049 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc37f0a8a064fa388608e88d86194c2, State: Running, Role: LEADER
I20260201 03:03:09.971863 26049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } }
I20260201 03:03:09.972353 26044 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:09.973505 26051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cc37f0a8a064fa388608e88d86194c2. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } }
I20260201 03:03:09.973624 26051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:09.973956 26058 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:03:09.974006 26050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } }
I20260201 03:03:09.974627 26050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:09.976836 26058 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:03:09.982340 26058 catalog_manager.cc:1357] Generated new cluster ID: ab56833fde3947a39e0b6397cf47f13e
I20260201 03:03:09.982422 26058 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:03:10.019632 26058 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:03:10.020620 26058 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:03:10.029980 26058 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2: Generated new TSK 0
I20260201 03:03:10.030740 26058 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
I20260201 03:03:10.078787 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:40003
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:10.188820 26072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:10.189074 26072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:10.189132 26072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:10.192410 26072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:10.192478 26072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:10.192566 26072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:03:10.197000 26072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:40003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:10.198063 26072 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:10.198880 26072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:10.205457 26077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:10.205458 26080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:10.205479 26078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:10.205770 26072 server_base.cc:1060] running on GCE node
I20260201 03:03:10.206123 26072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:10.206691 26072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:10.207835 26072 hybrid_clock.cc:648] HybridClock initialized: now 1769914990207823 us; error 27 us; skew 500 ppm
Feb 01 03:03:10 dist-test-slave-gf59 krb5kdc[25960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914990, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:10.210388 26072 init.cc:377] Logged in from keytab as oryx/127.18.203.193@KRBTEST.COM (short username oryx)
I20260201 03:03:10.211436 26072 webserver.cc:492] Webserver started at http://127.18.203.193:41329/ using document root <none> and password file <none>
I20260201 03:03:10.212013 26072 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:10.212067 26072 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:10.212244 26072 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:10.213905 26072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "90f106a46c90471c960543c8320a0242"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "6f65505b129bed208eb3194b2661fa2c"
server_key_iv: "fd0e3606a2b363b24e9e06ca7299b26e"
server_key_version: "encryptionkey@0"
I20260201 03:03:10.214376 26072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "90f106a46c90471c960543c8320a0242"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "6f65505b129bed208eb3194b2661fa2c"
server_key_iv: "fd0e3606a2b363b24e9e06ca7299b26e"
server_key_version: "encryptionkey@0"
I20260201 03:03:10.217695 26072 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260201 03:03:10.219838 26087 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:10.220806 26072 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:10.220906 26072 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "90f106a46c90471c960543c8320a0242"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "6f65505b129bed208eb3194b2661fa2c"
server_key_iv: "fd0e3606a2b363b24e9e06ca7299b26e"
server_key_version: "encryptionkey@0"
I20260201 03:03:10.220988 26072 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:10.246951 26072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:10.250684 26072 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:10.250898 26072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:10.251588 26072 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:10.252733 26072 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:10.252800 26072 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:10.252858 26072 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:10.252887 26072 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:10.261826 26072 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:42487
I20260201 03:03:10.261854 26200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:42487 every 8 connection(s)
I20260201 03:03:10.262881 26072 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:03:10.265467 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 26072
I20260201 03:03:10.265595 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:03:10.265833 19247 external_mini_cluster.cc:1468] Setting key 454f7a7138b1c70aa49933610c4bd006
Feb 01 03:03:10 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914990, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.18.203.194@KRBTEST.COM; defaulting to no policy
I20260201 03:03:10.275801 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.264651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:48021 (local address 127.18.203.254:40003)
0201 03:03:10.264781 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:10.264784 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:10.265481 (+ 697us) server_negotiation.cc:408] Connection header received
0201 03:03:10.266426 (+ 945us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:10.266430 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:10.266489 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:10.266559 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:10.268405 (+ 1846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.268909 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.269599 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.269771 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:10.272117 (+ 2346us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:10.272144 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:10.272148 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:10.272183 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:10.273746 (+ 1563us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:10.274285 (+ 539us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:10.274290 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:10.274292 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:10.274345 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:10.274618 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:10.274622 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:10.274624 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:10.274810 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:10.274899 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:10.275453 (+ 554us) server_negotiation.cc:300] Negotiation successful
0201 03:03:10.275603 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260201 03:03:10.276470 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.264875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.193:48021)
0201 03:03:10.265331 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:10.265367 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:10.266201 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:10.266732 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:10.266741 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:10.267252 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:10.268256 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.268271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.269051 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.269056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:10.269492 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.269500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.269681 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:10.270248 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:10.270293 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:10.271953 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:10.273862 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:10.273867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:10.273878 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:10.274124 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:10.274430 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:10.274433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:10.274436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:10.274533 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:10.274923 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:10.274931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:10.275202 (+ 271us) client_negotiation.cc:769] Sending connection context
0201 03:03:10.275403 (+ 201us) client_negotiation.cc:240] Negotiation successful
0201 03:03:10.275725 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":100,"threads_started":1}
I20260201 03:03:10.277786 26201 heartbeater.cc:344] Connected to a master server at 127.18.203.254:40003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.194@KRBTEST.COM" created.
I20260201 03:03:10.278069 26201 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:10.278672 26201 heartbeater.cc:507] Master 127.18.203.254:40003 requested a full tablet report, sending...
I20260201 03:03:10.280311 26008 ts_manager.cc:194] Registered new tserver with Master: 90f106a46c90471c960543c8320a0242 (127.18.203.193:42487)
I20260201 03:03:10.281527 26008 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:48021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
I20260201 03:03:10.315378 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:40003
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:10.422236 26208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:10.422477 26208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:10.422533 26208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:10.425858 26208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:10.425930 26208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:10.426010 26208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:03:10.430143 26208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:40003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:10.431174 26208 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:10.432031 26208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:10.438541 26214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:10.438555 26216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:10.438702 26213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:10.438884 26208 server_base.cc:1060] running on GCE node
I20260201 03:03:10.439277 26208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:10.439826 26208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:10.440969 26208 hybrid_clock.cc:648] HybridClock initialized: now 1769914990440959 us; error 28 us; skew 500 ppm
Feb 01 03:03:10 dist-test-slave-gf59 krb5kdc[25960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914990, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:10.443497 26208 init.cc:377] Logged in from keytab as oryx/127.18.203.194@KRBTEST.COM (short username oryx)
I20260201 03:03:10.444561 26208 webserver.cc:492] Webserver started at http://127.18.203.194:38785/ using document root <none> and password file <none>
I20260201 03:03:10.445084 26208 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:10.445133 26208 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:10.445317 26208 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:10.446957 26208 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "94bfc37fa718472db25e38cd9ad2366f"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "79579f45058f1e63b1888ca3ddbbbae3"
server_key_iv: "07b1c9f6f3d73492b1d79cc576fcf7ea"
server_key_version: "encryptionkey@0"
I20260201 03:03:10.447419 26208 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "94bfc37fa718472db25e38cd9ad2366f"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "79579f45058f1e63b1888ca3ddbbbae3"
server_key_iv: "07b1c9f6f3d73492b1d79cc576fcf7ea"
server_key_version: "encryptionkey@0"
I20260201 03:03:10.450657 26208 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260201 03:03:10.452858 26223 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:10.453836 26208 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:10.453938 26208 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "94bfc37fa718472db25e38cd9ad2366f"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "79579f45058f1e63b1888ca3ddbbbae3"
server_key_iv: "07b1c9f6f3d73492b1d79cc576fcf7ea"
server_key_version: "encryptionkey@0"
I20260201 03:03:10.454032 26208 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:10.467216 26208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:10.470153 26208 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:10.470345 26208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:10.470908 26208 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:10.471917 26208 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:10.471972 26208 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:10.472033 26208 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:10.472062 26208 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:10.480989 26208 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:45423
I20260201 03:03:10.481036 26336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:45423 every 8 connection(s)
I20260201 03:03:10.481985 26208 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
Feb 01 03:03:10 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914990, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:10.491526 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 26208
I20260201 03:03:10.491654 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:03:10.491886 19247 external_mini_cluster.cc:1468] Setting key 537db56f2fa534499ba2a689f79190c9
I20260201 03:03:10.494091 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.483726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:51145 (local address 127.18.203.254:40003)
0201 03:03:10.483853 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:10.483857 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:10.484603 (+ 746us) server_negotiation.cc:408] Connection header received
0201 03:03:10.485458 (+ 855us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:10.485462 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:10.485522 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:10.485596 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:10.487288 (+ 1692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.487840 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.488463 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.488650 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:10.490732 (+ 2082us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:10.490752 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:10.490754 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:10.490782 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:10.492151 (+ 1369us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:10.492679 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:10.492682 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:10.492684 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:10.492734 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:10.492975 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:10.492999 (+ 24us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:10.493001 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:10.493183 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:10.493287 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:10.493836 (+ 549us) server_negotiation.cc:300] Negotiation successful
0201 03:03:10.493953 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260201 03:03:10.494875 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.484046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.194:51145)
0201 03:03:10.484463 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:10.484500 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:10.485255 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:10.485764 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:10.485773 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:10.486182 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:10.487131 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.487146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.487956 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.487959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:10.488351 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.488358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.488572 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:10.489131 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:10.489157 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:10.490577 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:10.492266 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:10.492270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:10.492280 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:10.492582 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:10.492818 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:10.492820 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:10.492822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:10.492905 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:10.493295 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:10.493302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:10.493586 (+ 284us) client_negotiation.cc:769] Sending connection context
0201 03:03:10.493818 (+ 232us) client_negotiation.cc:240] Negotiation successful
0201 03:03:10.494090 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
I20260201 03:03:10.495992 26337 heartbeater.cc:344] Connected to a master server at 127.18.203.254:40003
I20260201 03:03:10.496268 26337 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:10.496771 26337 heartbeater.cc:507] Master 127.18.203.254:40003 requested a full tablet report, sending...
I20260201 03:03:10.497735 26008 ts_manager.cc:194] Registered new tserver with Master: 94bfc37fa718472db25e38cd9ad2366f (127.18.203.194:45423)
I20260201 03:03:10.498304 26008 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:51145
WARNING: no policy specified for oryx/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
I20260201 03:03:10.540696 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:40003
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:10.649611 26344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:10.649864 26344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:10.649931 26344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:10.653192 26344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:10.653261 26344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:10.653353 26344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:03:10.657498 26344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:40003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:10.658581 26344 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:10.659463 26344 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:10.666256 26350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:10.666260 26352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:10.666256 26349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:10.666676 26344 server_base.cc:1060] running on GCE node
I20260201 03:03:10.667065 26344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:10.667658 26344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:10.668802 26344 hybrid_clock.cc:648] HybridClock initialized: now 1769914990668782 us; error 40 us; skew 500 ppm
Feb 01 03:03:10 dist-test-slave-gf59 krb5kdc[25960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914990, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:10.671656 26344 init.cc:377] Logged in from keytab as oryx/127.18.203.195@KRBTEST.COM (short username oryx)
I20260201 03:03:10.672842 26344 webserver.cc:492] Webserver started at http://127.18.203.195:39745/ using document root <none> and password file <none>
I20260201 03:03:10.673501 26344 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:10.673559 26344 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:10.673764 26344 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:10.675586 26344 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "e8a481e12029469aa778691177e5ab15"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "d998ce425b607a1096be5b20054ae944"
server_key_iv: "542ced3cb0774ee147657ade6266df60"
server_key_version: "encryptionkey@0"
I20260201 03:03:10.676108 26344 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "e8a481e12029469aa778691177e5ab15"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "d998ce425b607a1096be5b20054ae944"
server_key_iv: "542ced3cb0774ee147657ade6266df60"
server_key_version: "encryptionkey@0"
I20260201 03:03:10.679574 26344 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260201 03:03:10.681856 26359 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:10.683022 26344 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:10.683138 26344 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "e8a481e12029469aa778691177e5ab15"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "d998ce425b607a1096be5b20054ae944"
server_key_iv: "542ced3cb0774ee147657ade6266df60"
server_key_version: "encryptionkey@0"
I20260201 03:03:10.683231 26344 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:10.694234 26344 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:10.697065 26344 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:10.697252 26344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:10.697801 26344 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:10.698704 26344 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:10.698753 26344 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:10.698804 26344 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:10.698829 26344 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:10.707744 26344 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:38661
I20260201 03:03:10.707773 26472 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:38661 every 8 connection(s)
I20260201 03:03:10.708724 26344 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
Feb 01 03:03:10 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914990, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:10.717468 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 26344
I20260201 03:03:10.717585 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:03:10.717818 19247 external_mini_cluster.cc:1468] Setting key f3b2e468714a503abc94710a2f60c36e
I20260201 03:03:10.721169 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.710458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:58229 (local address 127.18.203.254:40003)
0201 03:03:10.710610 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:10.710615 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:10.711312 (+ 697us) server_negotiation.cc:408] Connection header received
0201 03:03:10.712277 (+ 965us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:10.712281 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:10.712338 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:10.712429 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:10.714156 (+ 1727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.714852 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.715489 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.715679 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:10.717815 (+ 2136us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:10.717836 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:10.717838 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:10.717866 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:10.719205 (+ 1339us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:10.719779 (+ 574us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:10.719782 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:10.719785 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:10.719833 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:10.720125 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:10.720129 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:10.720131 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:10.720288 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:10.720374 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:10.720876 (+ 502us) server_negotiation.cc:300] Negotiation successful
0201 03:03:10.721009 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260201 03:03:10.721781 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.710748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.195:58229)
0201 03:03:10.711173 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:10.711209 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:10.712015 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:10.712576 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:10.712585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:10.712991 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:10.713973 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.713988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.714964 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.714967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:10.715369 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.715376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.715557 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:10.716151 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:10.716176 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:10.717661 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:10.719325 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:10.719331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:10.719343 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:10.719688 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:10.719923 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:10.719926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:10.719929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:10.720029 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:10.720384 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:10.720391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:10.720626 (+ 235us) client_negotiation.cc:769] Sending connection context
0201 03:03:10.720811 (+ 185us) client_negotiation.cc:240] Negotiation successful
0201 03:03:10.721086 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":97,"threads_started":1}
I20260201 03:03:10.722750 26473 heartbeater.cc:344] Connected to a master server at 127.18.203.254:40003
I20260201 03:03:10.722995 26473 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:10.723482 26473 heartbeater.cc:507] Master 127.18.203.254:40003 requested a full tablet report, sending...
I20260201 03:03:10.724567 26008 ts_manager.cc:194] Registered new tserver with Master: e8a481e12029469aa778691177e5ab15 (127.18.203.195:38661)
I20260201 03:03:10.725126 26008 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:58229
I20260201 03:03:10.732120 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.203.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.203.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.203.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.203.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
I20260201 03:03:10.778046 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.253:38411
--webserver_interface=127.18.203.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:40003,127.18.203.253:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:10.887305 26480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:10.887542 26480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:10.887594 26480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:10.890799 26480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:10.890862 26480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:10.890883 26480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:10.890903 26480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:10.890930 26480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:03:10.895043 26480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:40003,127.18.203.253:38411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.253:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.203.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
W20260201 03:03:10.895212 26480 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.18.203.254:40003,127.18.203.253:38411'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260201 03:03:10.896098 26480 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:10.896889 26480 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:10.902491 26486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:10.902493 26485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:10.902571 26480 server_base.cc:1060] running on GCE node
W20260201 03:03:10.902496 26488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:10.903192 26480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:10.904240 26480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:10.905386 26480 hybrid_clock.cc:648] HybridClock initialized: now 1769914990905364 us; error 47 us; skew 500 ppm
Feb 01 03:03:10 dist-test-slave-gf59 krb5kdc[25960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914990, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:10.907954 26480 init.cc:377] Logged in from keytab as oryx/127.18.203.253@KRBTEST.COM (short username oryx)
I20260201 03:03:10.909015 26480 webserver.cc:492] Webserver started at http://127.18.203.253:38209/ using document root <none> and password file <none>
I20260201 03:03:10.909569 26480 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:10.909621 26480 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:10.909811 26480 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:10.911561 26480 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data/instance:
uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "2b872eeeead1d9b95f7bf856929f1f8a"
server_key_iv: "93b553a77af88a8295add3f074b0b684"
server_key_version: "encryptionkey@0"
I20260201 03:03:10.912071 26480 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/wal/instance:
uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "2b872eeeead1d9b95f7bf856929f1f8a"
server_key_iv: "93b553a77af88a8295add3f074b0b684"
server_key_version: "encryptionkey@0"
I20260201 03:03:10.915380 26480 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260201 03:03:10.917526 26495 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:10.918478 26480 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:10.918582 26480 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/wal
uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "2b872eeeead1d9b95f7bf856929f1f8a"
server_key_iv: "93b553a77af88a8295add3f074b0b684"
server_key_version: "encryptionkey@0"
I20260201 03:03:10.918663 26480 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:10.937755 26480 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:10.940649 26480 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:10.940825 26480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:10.947768 26480 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.253:38411
I20260201 03:03:10.947767 26547 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.253:38411 every 8 connection(s)
I20260201 03:03:10.948740 26480 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data/info.pb
I20260201 03:03:10.951399 26548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:10.954576 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 26480
I20260201 03:03:10.955258 26548 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } has no permanent_uuid. Determining permanent_uuid...
Feb 01 03:03:10 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.203.253@KRBTEST.COM
Feb 01 03:03:10 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914990, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.253@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:10.966485 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.956933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.18.203.254:40003)
0201 03:03:10.957071 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:10.957075 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:10.957409 (+ 334us) server_negotiation.cc:408] Connection header received
0201 03:03:10.957661 (+ 252us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:10.957665 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:10.957721 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:10.957788 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:10.959207 (+ 1419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.959960 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.960770 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.960975 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:10.963589 (+ 2614us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:10.963648 (+ 59us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:10.963652 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:10.963682 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:10.964830 (+ 1148us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:10.965253 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:10.965256 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:10.965257 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:10.965311 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:10.965545 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:10.965548 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:10.965550 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:10.965729 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:10.965816 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:10.966177 (+ 361us) server_negotiation.cc:300] Negotiation successful
0201 03:03:10.966318 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260201 03:03:10.967173 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.955993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.18.203.253:38411)
0201 03:03:10.956429 (+ 436us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:10.956440 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:10.956477 (+ 37us) server_negotiation.cc:408] Connection header received
0201 03:03:10.957123 (+ 646us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:10.957145 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:10.957497 (+ 352us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:10.958392 (+ 895us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:10.958769 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.959652 (+ 883us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.960391 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.960672 (+ 281us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:10.963111 (+ 2439us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:10.963133 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:10.963146 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:10.963172 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:10.964959 (+ 1787us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:10.965341 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:10.965346 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:10.965347 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:10.965410 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:10.965619 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:10.965622 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:10.965624 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:10.965945 (+ 321us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:10.966114 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:10.966311 (+ 197us) server_negotiation.cc:300] Negotiation successful
0201 03:03:10.966481 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":298,"thread_start_us":152,"threads_started":1}
I20260201 03:03:10.967176 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.956869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:58604)
0201 03:03:10.957289 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:10.957327 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:10.957498 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:10.957843 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:10.957853 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:10.958240 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:10.959057 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.959069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.960096 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.960100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:10.960635 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.960646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.961140 (+ 494us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:10.961672 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:10.961709 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:10.963458 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:10.964934 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:10.964943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:10.964952 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:10.965160 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:10.965391 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:10.965394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:10.965396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:10.965466 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:10.965825 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:10.965833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:10.965953 (+ 120us) client_negotiation.cc:769] Sending connection context
0201 03:03:10.966133 (+ 180us) client_negotiation.cc:240] Negotiation successful
0201 03:03:10.966416 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"spinlock_wait_cycles":12032,"thread_start_us":84,"threads_started":1}
I20260201 03:03:10.968603 26548 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } has no permanent_uuid. Determining permanent_uuid...
Feb 01 03:03:10 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914990, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.253@KRBTEST.COM for oryx/127.18.203.253@KRBTEST.COM
I20260201 03:03:10.975502 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.969134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.18.203.253:38411)
0201 03:03:10.969276 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:10.969280 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:10.969359 (+ 79us) server_negotiation.cc:408] Connection header received
0201 03:03:10.969454 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:10.969457 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:10.969503 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:10.969567 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:10.970039 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.970497 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.971041 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.971226 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:10.972883 (+ 1657us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:10.972897 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:10.972899 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:10.972919 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:10.974237 (+ 1318us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:10.974552 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:10.974556 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:10.974558 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:10.974595 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:10.974782 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:10.974785 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:10.974787 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:10.974925 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:10.975004 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:10.975233 (+ 229us) server_negotiation.cc:300] Negotiation successful
0201 03:03:10.975363 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":57}
I20260201 03:03:10.975502 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.969083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.253:38411 (local address 127.0.0.1:54638)
0201 03:03:10.969276 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:10.969291 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:10.969382 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:10.969585 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:10.969588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:10.969755 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:10.969929 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.969935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.970584 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.970587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:10.970926 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.970931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.971052 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:10.971515 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:10.971527 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:10.972782 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:10.974322 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:10.974326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:10.974328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:10.974475 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:10.974665 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:10.974668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:10.974670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:10.974717 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:10.975002 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:10.975005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:10.975092 (+ 87us) client_negotiation.cc:769] Sending connection context
0201 03:03:10.975175 (+ 83us) client_negotiation.cc:240] Negotiation successful
0201 03:03:10.975308 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260201 03:03:10.978632 26548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: Bootstrap starting.
I20260201 03:03:10.981328 26548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:10.982079 26548 log.cc:826] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:10.983913 26548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: No bootstrap required, opened a new log
I20260201 03:03:10.986335 26548 raft_consensus.cc:359] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } }
I20260201 03:03:10.986506 26548 raft_consensus.cc:385] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:10.986559 26548 raft_consensus.cc:740] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a7a00d1b784a3fa9cd98e967c8eed4, State: Initialized, Role: FOLLOWER
I20260201 03:03:10.986984 26548 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } }
I20260201 03:03:10.987462 26554 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } } }
I20260201 03:03:10.987567 26554 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: This master's current role is: FOLLOWER
I20260201 03:03:10.988091 26548 sys_catalog.cc:565] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: configured and running, proceeding with master startup.
W20260201 03:03:10.992408 26565 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260201 03:03:10.992497 26565 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260201 03:03:10.994086 26480 master_runner.cc:428] Detected that this master d3a7a00d1b784a3fa9cd98e967c8eed4 is joining an existing cluster
I20260201 03:03:10.994136 26480 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.203.253:38411
I20260201 03:03:11.003720 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.996853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.18.203.253:38411)
0201 03:03:10.996966 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:10.996970 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:10.996983 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:10.997060 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:10.997062 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:10.997100 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:10.997178 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:10.998334 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.998993 (+ 659us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.999857 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.000037 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.000811 (+ 774us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.000825 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.000827 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.000848 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.002166 (+ 1318us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.002595 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.002598 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.002599 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.002636 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.002878 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.002881 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.002882 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.003045 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.003129 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.003396 (+ 267us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.003549 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":45}
I20260201 03:03:11.003736 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.996615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.253:38411 (local address 127.0.0.1:54650)
0201 03:03:10.996894 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:10.996907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:10.996983 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:10.997204 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:10.997208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:10.997904 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:10.998199 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.998208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.999111 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.999114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:10.999715 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.999724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.999855 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.000297 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:11.000313 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:11.000672 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:11.002282 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.002286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.002289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.002503 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.002707 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.002711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.002713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.002768 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.003133 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:11.003137 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:11.003226 (+ 89us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.003342 (+ 116us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.003485 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
I20260201 03:03:11.005306 26554 raft_consensus.cc:493] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:03:11.005470 26554 raft_consensus.cc:515] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } }
I20260201 03:03:11.006475 26554 leader_election.cc:290] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc37f0a8a064fa388608e88d86194c2 (127.18.203.254:40003)
I20260201 03:03:11.006991 26018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc37f0a8a064fa388608e88d86194c2" is_pre_election: true
I20260201 03:03:11.007669 26499 leader_election.cc:304] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: d3a7a00d1b784a3fa9cd98e967c8eed4; no voters: 4cc37f0a8a064fa388608e88d86194c2
I20260201 03:03:11.007929 26554 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260201 03:03:11.019685 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.996055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:58612)
0201 03:03:10.996341 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:10.996354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:10.996461 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:10.996733 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:10.996736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:10.996989 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:10.997307 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.997314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.998553 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.998557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:10.999055 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:10.999064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.999170 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:10.999726 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:10.999741 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:11.016054 (+ 16313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:11.018103 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.018108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.018110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.018329 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.018605 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.018609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.018611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.018659 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.019038 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:11.019042 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:11.019153 (+ 111us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.019256 (+ 103us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.019417 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":113,"threads_started":1}
I20260201 03:03:11.019670 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:10.996216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.18.203.254:40003)
0201 03:03:10.996365 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:10.996370 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:10.996431 (+ 61us) server_negotiation.cc:408] Connection header received
0201 03:03:10.996552 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:10.996556 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:10.996620 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:10.996692 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:10.997422 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.998310 (+ 888us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:10.999169 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:10.999358 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.016289 (+ 16931us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.016316 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.016319 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.016360 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.017957 (+ 1597us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.018454 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.018458 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.018459 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.018515 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.018756 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.018759 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.018760 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.018943 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.019040 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.019328 (+ 288us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.019450 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Feb 01 03:03:11 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914989, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.254@KRBTEST.COM for oryx/127.18.203.253@KRBTEST.COM
WARNING: no policy specified for oryx/127.18.203.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.252@KRBTEST.COM" created.
I20260201 03:03:11.034708 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.024821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.18.203.253:38411)
0201 03:03:11.024955 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.024961 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.025327 (+ 366us) server_negotiation.cc:408] Connection header received
0201 03:03:11.025554 (+ 227us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.025557 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.025650 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.025731 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:11.026735 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.027416 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.028256 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.028453 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.030923 (+ 2470us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.030946 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.030950 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.030982 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.032754 (+ 1772us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.033333 (+ 579us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.033336 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.033337 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.033388 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.033616 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.033619 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.033621 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.033818 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.033908 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.034395 (+ 487us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.034529 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260201 03:03:11.035310 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.024798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.253:38411 (local address 127.0.0.1:54658)
0201 03:03:11.025198 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.025241 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.025400 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.025771 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.025777 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.026298 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:11.026597 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.026606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.027536 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.027541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.028130 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.028140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.028256 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.028838 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:11.028854 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:11.030778 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:11.032890 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.032900 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.032906 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.033227 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.033474 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.033478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.033480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.033537 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.033923 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:11.033931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:11.034069 (+ 138us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.034933 (+ 864us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.035129 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":116,"threads_started":1}
I20260201 03:03:11.038062 26008 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.18.203.253:38411: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4"
member_type: NON_VOTER
last_known_addr {
host: "127.18.203.253"
port: 38411
}
attrs {
promote: true
}
}
dest_uuid: "4cc37f0a8a064fa388608e88d86194c2"
cas_config_opid_index: -1
I20260201 03:03:11.038771 26008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true } }
I20260201 03:03:11.040771 26577 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d3a7a00d1b784a3fa9cd98e967c8eed4 (127.18.203.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true } } }
I20260201 03:03:11.041674 26522 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:11.043228 26522 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 FOLLOWER]: Refusing update from remote peer 4cc37f0a8a064fa388608e88d86194c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260201 03:03:11.044656 26577 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.18.203.253:38411
I20260201 03:03:11.044647 26578 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:11.045018 26577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true } } }
I20260201 03:03:11.045107 26577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:11.045269 26480 master.cc:561] Master@127.18.203.253:38411 shutting down...
W20260201 03:03:11.047405 25995 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260201 03:03:11.048604 25995 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 -> Peer d3a7a00d1b784a3fa9cd98e967c8eed4 (127.18.203.253:38411): Couldn't send request to peer d3a7a00d1b784a3fa9cd98e967c8eed4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.203.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
I20260201 03:03:11.050104 26480 raft_consensus.cc:2243] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:11.050379 26480 raft_consensus.cc:2272] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:11.050485 26480 tablet_replica.cc:333] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: stopping tablet replica
I20260201 03:03:11.054698 26480 master.cc:583] Master@127.18.203.253:38411 shutdown complete.
I20260201 03:03:11.054751 26480 master_runner.cc:305] Clearing existing system tablet
I20260201 03:03:11.054972 26480 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:11.055028 26480 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:11.056321 26480 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260201 03:03:11.057224 26585 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:11.057471 26480 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:11.057657 26480 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260201 03:03:11.057720 26480 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/wal
uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "2b872eeeead1d9b95f7bf856929f1f8a"
server_key_iv: "93b553a77af88a8295add3f074b0b684"
server_key_version: "encryptionkey@0"
WARNING: no policy specified for HTTP/127.18.203.252@KRBTEST.COM; defaulting to no policy
I20260201 03:03:11.059864 26480 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.252@KRBTEST.COM" created.
I20260201 03:03:11.062662 26480 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260201 03:03:11.062754 26480 log.cc:1199] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260201 03:03:11.063009 26480 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: Deleting consensus metadata
I20260201 03:03:11.063170 26480 master_runner.cc:315] Copying system tablet from 127.18.203.254:40003
I20260201 03:03:11.064414 26480 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: tablet copy: Beginning tablet copy session from remote peer at address 127.18.203.254:40003
I20260201 03:03:11.073498 26592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.065019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:58622)
0201 03:03:11.065389 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.065404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.065791 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.066168 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.066172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.066392 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:11.066639 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.066647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.067794 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.067797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.068419 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.068428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.068888 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.069484 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:11.069502 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:11.069904 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:11.071799 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.071803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.071806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.072031 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.072313 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.072318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.072320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.072378 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.072767 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:11.072771 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:11.072896 (+ 125us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.073070 (+ 174us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.073310 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":95,"threads_started":1}
I20260201 03:03:11.073484 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.065096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.18.203.254:40003)
0201 03:03:11.065242 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.065247 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.065699 (+ 452us) server_negotiation.cc:408] Connection header received
0201 03:03:11.065913 (+ 214us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.065917 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.065974 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.066026 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:11.066790 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.067599 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.068553 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.068757 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.070046 (+ 1289us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.070067 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.070070 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.070108 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.071673 (+ 1565us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.072132 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.072136 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.072137 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.072206 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.072470 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.072474 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.072476 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.072655 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.072758 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.073143 (+ 385us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.073289 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260201 03:03:11.074220 26028 tablet_copy_service.cc:140] P 4cc37f0a8a064fa388608e88d86194c2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d3a7a00d1b784a3fa9cd98e967c8eed4 ({username='oryx', principal='oryx/127.18.203.253@KRBTEST.COM'} at 127.0.0.1:58622)
I20260201 03:03:11.074412 26028 tablet_copy_service.cc:161] P 4cc37f0a8a064fa388608e88d86194c2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d3a7a00d1b784a3fa9cd98e967c8eed4 at {username='oryx', principal='oryx/127.18.203.253@KRBTEST.COM'} at 127.0.0.1:58622: session id = d3a7a00d1b784a3fa9cd98e967c8eed4-00000000000000000000000000000000
I20260201 03:03:11.076056 26028 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2: Tablet Copy: opened 0 blocks and 1 log segments
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.203.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab.
I20260201 03:03:11.077747 26480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260201 03:03:11.078353 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.252:35217
--webserver_interface=127.18.203.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:40003,127.18.203.253:38411,127.18.203.252:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260201 03:03:11.079965 26480 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: tablet copy: Starting download of 0 data blocks...
I20260201 03:03:11.080142 26480 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: tablet copy: Starting download of 1 WAL segments...
I20260201 03:03:11.082201 26480 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260201 03:03:11.084517 26028 tablet_copy_service.cc:342] P 4cc37f0a8a064fa388608e88d86194c2: Request end of tablet copy session d3a7a00d1b784a3fa9cd98e967c8eed4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.203.253@KRBTEST.COM'} at 127.0.0.1:58622
I20260201 03:03:11.084699 26028 tablet_copy_service.cc:434] P 4cc37f0a8a064fa388608e88d86194c2: ending tablet copy session d3a7a00d1b784a3fa9cd98e967c8eed4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d3a7a00d1b784a3fa9cd98e967c8eed4
W20260201 03:03:11.089762 26480 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260201 03:03:11.092420 26480 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:11.094074 26594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:11.094201 26595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:11.096715 26597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:11.096954 26480 server_base.cc:1060] running on GCE node
I20260201 03:03:11.097112 26480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:11.097432 26480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:11.098579 26480 hybrid_clock.cc:648] HybridClock initialized: now 1769914991098562 us; error 36 us; skew 500 ppm
Feb 01 03:03:11 dist-test-slave-gf59 krb5kdc[25960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914991, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:11.100399 26480 init.cc:377] Logged in from keytab as oryx/127.18.203.253@KRBTEST.COM (short username oryx)
I20260201 03:03:11.101033 26480 webserver.cc:492] Webserver started at http://127.18.203.253:35643/ using document root <none> and password file <none>
I20260201 03:03:11.101248 26480 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:11.101286 26480 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:11.102669 26480 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:11.103858 26604 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:11.104151 26480 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:11.104242 26480 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/wal
uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "2b872eeeead1d9b95f7bf856929f1f8a"
server_key_iv: "93b553a77af88a8295add3f074b0b684"
server_key_version: "encryptionkey@0"
I20260201 03:03:11.104326 26480 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:11.111637 26480 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:11.113677 26480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:11.118257 26480 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.253:38411
I20260201 03:03:11.118431 26656 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.253:38411 every 8 connection(s)
I20260201 03:03:11.118863 26480 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-1/data/info.pb
I20260201 03:03:11.120584 26657 sys_catalog.cc:263] Verifying existing consensus state
I20260201 03:03:11.121639 26657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: Bootstrap starting.
I20260201 03:03:11.131644 26657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260201 03:03:11.132119 26657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: Bootstrap complete.
I20260201 03:03:11.132938 26657 raft_consensus.cc:359] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true } }
I20260201 03:03:11.133195 26657 raft_consensus.cc:740] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3a7a00d1b784a3fa9cd98e967c8eed4, State: Initialized, Role: LEARNER
I20260201 03:03:11.133363 26657 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true } }
I20260201 03:03:11.133838 26660 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true } } }
I20260201 03:03:11.133953 26657 sys_catalog.cc:565] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:11.133983 26660 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: This master's current role is: LEARNER
I20260201 03:03:11.137537 26671 catalog_manager.cc:1269] Loaded cluster ID: ab56833fde3947a39e0b6397cf47f13e
I20260201 03:03:11.137596 26671 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: loading cluster ID for follower catalog manager: success
I20260201 03:03:11.139685 26671 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: acquiring CA information for follower catalog manager: success
I20260201 03:03:11.140981 26671 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260201 03:03:11.203212 26593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:11.203496 26593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:11.203555 26593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:11.207063 26593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:11.207144 26593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:11.207172 26593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:11.207196 26593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:11.207226 26593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:03:11.211740 26593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:40003,127.18.203.253:38411,127.18.203.252:35217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.252:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.203.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:11.213068 26593 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:11.214017 26593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:11.219874 26677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:11.219874 26676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:11.219874 26679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:11.220152 26593 server_base.cc:1060] running on GCE node
I20260201 03:03:11.220669 26593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:11.221580 26593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:11.222723 26593 hybrid_clock.cc:648] HybridClock initialized: now 1769914991222713 us; error 28 us; skew 500 ppm
Feb 01 03:03:11 dist-test-slave-gf59 krb5kdc[25960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914991, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:11.225459 26593 init.cc:377] Logged in from keytab as oryx/127.18.203.252@KRBTEST.COM (short username oryx)
I20260201 03:03:11.226514 26593 webserver.cc:492] Webserver started at http://127.18.203.252:41249/ using document root <none> and password file <none>
I20260201 03:03:11.227020 26593 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:11.227066 26593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:11.227227 26593 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:11.229017 26593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data/instance:
uuid: "0d13a9f0812f4ec58341619045702650"
format_stamp: "Formatted at 2026-02-01 03:03:11 on dist-test-slave-gf59"
server_key: "9b43439e4777084c0af46e8ce7966eb5"
server_key_iv: "6d4110b0ea4d46e89507977c37b8c526"
server_key_version: "encryptionkey@0"
I20260201 03:03:11.229470 26593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/wal/instance:
uuid: "0d13a9f0812f4ec58341619045702650"
format_stamp: "Formatted at 2026-02-01 03:03:11 on dist-test-slave-gf59"
server_key: "9b43439e4777084c0af46e8ce7966eb5"
server_key_iv: "6d4110b0ea4d46e89507977c37b8c526"
server_key_version: "encryptionkey@0"
I20260201 03:03:11.233023 26593 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260201 03:03:11.235527 26686 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:11.236670 26593 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:11.236778 26593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/wal
uuid: "0d13a9f0812f4ec58341619045702650"
format_stamp: "Formatted at 2026-02-01 03:03:11 on dist-test-slave-gf59"
server_key: "9b43439e4777084c0af46e8ce7966eb5"
server_key_iv: "6d4110b0ea4d46e89507977c37b8c526"
server_key_version: "encryptionkey@0"
I20260201 03:03:11.236863 26593 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:11.266075 26593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:11.268836 26593 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:11.269021 26593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:11.276876 26738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.252:35217 every 8 connection(s)
I20260201 03:03:11.276885 26593 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.252:35217
I20260201 03:03:11.277979 26593 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data/info.pb
I20260201 03:03:11.280822 26739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:11.283998 26201 heartbeater.cc:499] Master 127.18.203.254:40003 was elected leader, sending a full tablet report...
I20260201 03:03:11.284988 26739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } has no permanent_uuid. Determining permanent_uuid...
I20260201 03:03:11.286012 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 26593
Feb 01 03:03:11 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914991, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.252@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
Feb 01 03:03:11 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.203.252@KRBTEST.COM
I20260201 03:03:11.296801 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.287112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.18.203.254:40003)
0201 03:03:11.287225 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.287229 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.287870 (+ 641us) server_negotiation.cc:408] Connection header received
0201 03:03:11.288555 (+ 685us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.288558 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.288607 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.288680 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:11.289669 (+ 989us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.290160 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.290783 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.290966 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.293281 (+ 2315us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.293301 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.293303 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.293332 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.294802 (+ 1470us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.295267 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.295273 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.295275 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.295332 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.295585 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.295587 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.295588 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.295794 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.295886 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.296481 (+ 595us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.296614 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260201 03:03:11.297513 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.287173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:58636)
0201 03:03:11.287743 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.287782 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.288370 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.288737 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.288742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.289235 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:11.289526 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.289542 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.290263 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.290266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.290677 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.290684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.290846 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.291557 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:11.291581 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:11.293147 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:11.294916 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.294923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.294930 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.295168 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.295414 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.295416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.295418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.295498 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.295902 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:11.295908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:11.296230 (+ 322us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.296426 (+ 196us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.296728 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":143,"threads_started":1}
I20260201 03:03:11.297509 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.287173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.18.203.252:35217)
0201 03:03:11.287697 (+ 524us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.287708 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.287747 (+ 39us) server_negotiation.cc:408] Connection header received
0201 03:03:11.288358 (+ 611us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.288384 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.288615 (+ 231us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.288770 (+ 155us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:11.289586 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.290286 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.290927 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.291137 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.293685 (+ 2548us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.293707 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.293719 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.293743 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.295444 (+ 1701us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.295860 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.295866 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.295867 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.295927 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.296142 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.296144 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.296146 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.296320 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.296494 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.296659 (+ 165us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.296788 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"spinlock_wait_cycles":51456,"thread_start_us":143,"threads_started":1}
I20260201 03:03:11.298777 26739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } has no permanent_uuid. Determining permanent_uuid...
Feb 01 03:03:11 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914991, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.252@KRBTEST.COM for oryx/127.18.203.253@KRBTEST.COM
I20260201 03:03:11.306267 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.299286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.253:38411 (local address 127.0.0.1:54670)
0201 03:03:11.299453 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.299467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.299535 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.299865 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.299867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.300010 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:11.300210 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.300216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.301074 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.301077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.301477 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.301486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.301595 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.302067 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:11.302079 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:11.303307 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:11.305051 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.305055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.305057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.305216 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.305449 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.305453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.305455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.305508 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.305839 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:11.305841 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:11.305921 (+ 80us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.306001 (+ 80us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.306120 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260201 03:03:11.306286 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.299380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.18.203.253:38411)
0201 03:03:11.299674 (+ 294us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.299679 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.299693 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:03:11.299727 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.299729 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.299772 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.299848 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:11.300305 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.300989 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.301568 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.301691 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.303449 (+ 1758us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.303467 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.303470 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.303496 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.304944 (+ 1448us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.305308 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.305311 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.305312 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.305363 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.305597 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.305600 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.305602 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.305767 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.305844 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.306043 (+ 199us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.306149 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":94,"threads_started":1}
I20260201 03:03:11.306977 26739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } has no permanent_uuid. Determining permanent_uuid...
Feb 01 03:03:11 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914991, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.252@KRBTEST.COM for oryx/127.18.203.252@KRBTEST.COM
I20260201 03:03:11.313510 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.307496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.18.203.252:35217)
0201 03:03:11.307582 (+ 86us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.307586 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.307637 (+ 51us) server_negotiation.cc:408] Connection header received
0201 03:03:11.307768 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.307772 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.307803 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.307861 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:11.308299 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.308749 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.309275 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.309451 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.310935 (+ 1484us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.310952 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.310954 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.310974 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.312266 (+ 1292us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.312587 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.312590 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.312593 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.312628 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.312814 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.312817 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.312819 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.312955 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.313037 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.313249 (+ 212us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.313382 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":33}
I20260201 03:03:11.313510 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.307449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.0.0.1:33680)
0201 03:03:11.307565 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.307579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.307698 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.307884 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.307887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.308047 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:11.308202 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.308207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.308836 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.308839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.309178 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.309183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.309281 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.309696 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:11.309707 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:11.310831 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:11.312360 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.312363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.312366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.312509 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.312698 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.312701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.312703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.312748 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.313033 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:11.313036 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:11.313109 (+ 73us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.313192 (+ 83us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.313321 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260201 03:03:11.316381 26739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: Bootstrap starting.
I20260201 03:03:11.318715 26739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:11.319442 26739 log.cc:826] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:11.321300 26739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: No bootstrap required, opened a new log
I20260201 03:03:11.323812 26739 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } }
I20260201 03:03:11.323990 26739 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:11.324045 26739 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d13a9f0812f4ec58341619045702650, State: Initialized, Role: FOLLOWER
I20260201 03:03:11.324455 26739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } }
I20260201 03:03:11.324894 26745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } } }
I20260201 03:03:11.325007 26745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: This master's current role is: FOLLOWER
I20260201 03:03:11.325654 26739 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: configured and running, proceeding with master startup.
W20260201 03:03:11.329886 26756 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260201 03:03:11.329955 26756 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260201 03:03:11.333078 26593 master_runner.cc:428] Detected that this master 0d13a9f0812f4ec58341619045702650 is joining an existing cluster
I20260201 03:03:11.333132 26593 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.203.252:35217
I20260201 03:03:11.354996 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.351940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:58656)
0201 03:03:11.352081 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.352097 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.352194 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.352524 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.352528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.352545 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:11.352796 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.352804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.353539 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.353543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.354519 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.354529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.354648 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.354726 (+ 78us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.354778 (+ 52us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.354847 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260201 03:03:11.355273 26765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.351997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.18.203.254:40003)
0201 03:03:11.352327 (+ 330us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.352330 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.352341 (+ 11us) server_negotiation.cc:408] Connection header received
0201 03:03:11.352390 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.352393 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.352432 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.352503 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:11.352894 (+ 391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.353445 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.354646 (+ 1201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.354966 (+ 320us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.355041 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.355083 (+ 42us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.355147 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":150,"threads_started":1}
I20260201 03:03:11.358994 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.356378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.253:38411 (local address 127.0.0.1:54698)
0201 03:03:11.356505 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.356519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.356599 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.356981 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.356983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.356997 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:11.357142 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.357146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.358159 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.358162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.358713 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.358720 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.358808 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.358823 (+ 15us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.358856 (+ 33us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.358900 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260201 03:03:11.359742 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.356476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.18.203.253:38411)
0201 03:03:11.356739 (+ 263us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.356744 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.356759 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:03:11.356806 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.356809 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.356871 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.356976 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:11.357240 (+ 264us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.358053 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.358817 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.359308 (+ 491us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.359464 (+ 156us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.359517 (+ 53us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.359574 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260201 03:03:11.370719 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.335985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.0.0.1:33692)
0201 03:03:11.339817 (+ 3832us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.339831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.339932 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.340141 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.340145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.364022 (+ 23877us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:11.364665 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.364671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.365559 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.365562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.366124 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.366131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.366228 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.366841 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:11.366856 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:11.367296 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:11.369474 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.369477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.369479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.369661 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.369853 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.369856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.369858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.369899 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.370227 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:11.370230 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:11.370308 (+ 78us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.370396 (+ 88us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.370515 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3760,"spinlock_wait_cycles":17024,"thread_start_us":73,"threads_started":1}
I20260201 03:03:11.371223 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.335984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.18.203.252:35217)
0201 03:03:11.336119 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.336123 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.339908 (+ 3785us) server_negotiation.cc:408] Connection header received
0201 03:03:11.340003 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.340006 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.340048 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.340110 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:11.364761 (+ 24651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.365442 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.366463 (+ 1021us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.366644 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.367409 (+ 765us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.367423 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.367426 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.367446 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.369366 (+ 1920us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.369743 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.369746 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.369748 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.369782 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.369967 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.369969 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.369971 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.370120 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.370197 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.370986 (+ 789us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.371086 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260201 03:03:11.371517 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.335128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:58642)
0201 03:03:11.335877 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.335892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.336076 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.336636 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.336639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.364022 (+ 27383us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:11.364423 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.364434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.366163 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.366166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.366650 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.366659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.366770 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.367895 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:11.367910 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:11.368274 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:11.370184 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.370187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.370188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.370375 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.370626 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.370629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.370631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.370681 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.371076 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:11.371079 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:11.371152 (+ 73us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.371235 (+ 83us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.371364 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"spinlock_wait_cycles":9984,"thread_start_us":116,"threads_started":1}
I20260201 03:03:11.372062 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.335253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.18.203.254:40003)
0201 03:03:11.335423 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.335427 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.335998 (+ 571us) server_negotiation.cc:408] Connection header received
0201 03:03:11.336286 (+ 288us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.336293 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.336375 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.336485 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:11.365126 (+ 28641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.365836 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.366967 (+ 1131us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.367172 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.368384 (+ 1212us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.368400 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.368403 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.368429 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.370070 (+ 1641us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.370482 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.370486 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.370488 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.370537 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.370778 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.370782 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.370783 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.370955 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.371035 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.371815 (+ 780us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.371925 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260201 03:03:11.373718 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.335585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.253:38411 (local address 127.0.0.1:54684)
0201 03:03:11.336294 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.336307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.336400 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.336656 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.336660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.364297 (+ 27637us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:11.364524 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.364533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.365510 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.365514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.366090 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.366099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.366200 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.367500 (+ 1300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:11.367515 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:11.367897 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:11.369905 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.369908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.369911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.370115 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.372759 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.372762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.372765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.372832 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.373264 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:11.373267 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:11.373344 (+ 77us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.373452 (+ 108us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.373567 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":98,"threads_started":1}
I20260201 03:03:11.373837 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.335738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.18.203.253:38411)
0201 03:03:11.335877 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.335882 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.336406 (+ 524us) server_negotiation.cc:408] Connection header received
0201 03:03:11.336485 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.336489 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.336541 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.336629 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:11.364664 (+ 28035us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.365382 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.366825 (+ 1443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.367222 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.368029 (+ 807us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.368049 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.368052 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.368080 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.369760 (+ 1680us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.372607 (+ 2847us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.372611 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.372613 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.372664 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.372947 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.372951 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.372953 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.373136 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.373222 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.373562 (+ 340us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.373689 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Feb 01 03:03:11 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914989, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.254@KRBTEST.COM for oryx/127.18.203.252@KRBTEST.COM
I20260201 03:03:11.382712 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.375765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.0.0.1:33698)
0201 03:03:11.375899 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.375913 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.376007 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.376282 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.376285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.376438 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:11.376606 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.376612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.377223 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.377226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.377568 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.377574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.377666 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.378118 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:11.378130 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:11.379520 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:11.381610 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.381613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.381614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.381771 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.381973 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.381974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.381976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.382013 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.382340 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:11.382342 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:11.382419 (+ 77us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.382490 (+ 71us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.382594 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260201 03:03:11.382838 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.375817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.18.203.252:35217)
0201 03:03:11.375916 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.375920 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.375954 (+ 34us) server_negotiation.cc:408] Connection header received
0201 03:03:11.376141 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.376144 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.376186 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.376246 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:11.376707 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.377130 (+ 423us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.377661 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.377831 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.379687 (+ 1856us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.379704 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.379708 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.379733 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.381509 (+ 1776us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.381852 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.381856 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.381858 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.381904 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.382080 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.382083 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.382085 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.382266 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.382356 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.382549 (+ 193us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.382660 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260201 03:03:11.383694 26008 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.18.203.252:35217: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0d13a9f0812f4ec58341619045702650"
member_type: NON_VOTER
last_known_addr {
host: "127.18.203.252"
port: 35217
}
attrs {
promote: true
}
}
dest_uuid: "4cc37f0a8a064fa388608e88d86194c2"
cas_config_opid_index: 5
I20260201 03:03:11.384236 26008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } }
I20260201 03:03:11.385216 26580 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0d13a9f0812f4ec58341619045702650 (127.18.203.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:11.385787 26713 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:11.386688 26580 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.18.203.252:35217
I20260201 03:03:11.386859 26580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:11.386951 26580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:11.387465 26593 master.cc:561] Master@127.18.203.252:35217 shutting down...
I20260201 03:03:11.387595 26713 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 1 FOLLOWER]: Refusing update from remote peer 4cc37f0a8a064fa388608e88d86194c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260201 03:03:11.388464 26631 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 LEARNER]: Refusing update from remote peer 4cc37f0a8a064fa388608e88d86194c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260201 03:03:11.388903 26580 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:11.389226 26578 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260201 03:03:11.392885 25995 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 -> Peer 0d13a9f0812f4ec58341619045702650 (127.18.203.252:35217): Couldn't send request to peer 0d13a9f0812f4ec58341619045702650. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260201 03:03:11.392896 26593 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:11.392678 26631 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0d13a9f0812f4ec58341619045702650 (127.18.203.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:11.393158 26593 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:11.393253 26593 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: stopping tablet replica
I20260201 03:03:11.394877 26660 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cc37f0a8a064fa388608e88d86194c2. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:11.394989 26660 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: This master's current role is: LEARNER
I20260201 03:03:11.395076 26660 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: NON_VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: true } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:11.395133 26660 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: This master's current role is: LEARNER
I20260201 03:03:11.396030 26577 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2: attempting to promote NON_VOTER d3a7a00d1b784a3fa9cd98e967c8eed4 to VOTER
W20260201 03:03:11.396306 26008 master.cc:679] Remote error: unable to get registration information for peer 0d13a9f0812f4ec58341619045702650: Service unavailable: service kudu.master.MasterService not registered on Master
I20260201 03:03:11.396504 26577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } }
I20260201 03:03:11.397689 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.397337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.0.0.1:33714)
0201 03:03:11.397550 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.397608 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.252:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260201 03:03:11.397729 26631 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 LEARNER]: Refusing update from remote peer 4cc37f0a8a064fa388608e88d86194c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260201 03:03:11.398278 26580 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260201 03:03:11.398690 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.398173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.0.0.1:33728)
0201 03:03:11.398582 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.398629 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.252:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342}
W20260201 03:03:11.398907 25995 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 -> Peer 0d13a9f0812f4ec58341619045702650 (127.18.203.252:35217): Couldn't send request to peer 0d13a9f0812f4ec58341619045702650. Status: Network error: Client connection negotiation failed: client connection to 127.18.203.252:35217: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260201 03:03:11.399080 26593 master.cc:583] Master@127.18.203.252:35217 shutdown complete.
I20260201 03:03:11.399123 26593 master_runner.cc:305] Clearing existing system tablet
I20260201 03:03:11.399343 26593 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:11.399394 26593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:11.400277 26577 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d3a7a00d1b784a3fa9cd98e967c8eed4 (127.18.203.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:11.400703 26593 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:11.401458 26770 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:11.401600 26577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:11.401644 26593 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:11.401700 26577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:11.401826 26593 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:11.401829 26577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:11.401893 26577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:11.401870 26593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/wal
uuid: "0d13a9f0812f4ec58341619045702650"
format_stamp: "Formatted at 2026-02-01 03:03:11 on dist-test-slave-gf59"
server_key: "9b43439e4777084c0af46e8ce7966eb5"
server_key_iv: "6d4110b0ea4d46e89507977c37b8c526"
server_key_version: "encryptionkey@0"
I20260201 03:03:11.402523 26631 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d3a7a00d1b784a3fa9cd98e967c8eed4 (127.18.203.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:11.403848 26593 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:11.404021 26768 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:11.404182 26768 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: This master's current role is: FOLLOWER
I20260201 03:03:11.405632 26593 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260201 03:03:11.405704 26593 log.cc:1199] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260201 03:03:11.405934 26593 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: Deleting consensus metadata
I20260201 03:03:11.406090 26593 master_runner.cc:315] Copying system tablet from 127.18.203.254:40003
I20260201 03:03:11.407106 26593 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: tablet copy: Beginning tablet copy session from remote peer at address 127.18.203.254:40003
I20260201 03:03:11.413877 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.407714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.18.203.254:40003)
0201 03:03:11.407840 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.407843 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.407931 (+ 88us) server_negotiation.cc:408] Connection header received
0201 03:03:11.408063 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.408065 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.408109 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.408177 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:11.408817 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.409316 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.410102 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.410225 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.411142 (+ 917us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:11.411158 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:11.411161 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:11.411190 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:11.412543 (+ 1353us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.412956 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.412960 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.412962 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.413003 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:11.413211 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:11.413213 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:11.413214 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:11.413360 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:11.413428 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.413658 (+ 230us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.413750 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260201 03:03:11.413952 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.407574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:58658)
0201 03:03:11.407847 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.407862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.407977 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.408219 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.408222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.408455 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:11.408716 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.408725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.409408 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.409411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.410002 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.410011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.410118 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.410621 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:11.410637 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:11.411023 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:11.412642 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.412646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.412649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.412860 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.413083 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:11.413086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:11.413089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:11.413141 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:11.413428 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:11.413431 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:11.413523 (+ 92us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.413622 (+ 99us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.413760 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":110,"threads_started":1}
I20260201 03:03:11.414669 26028 tablet_copy_service.cc:140] P 4cc37f0a8a064fa388608e88d86194c2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0d13a9f0812f4ec58341619045702650 ({username='oryx', principal='oryx/127.18.203.252@KRBTEST.COM'} at 127.0.0.1:58658)
I20260201 03:03:11.414775 26028 tablet_copy_service.cc:161] P 4cc37f0a8a064fa388608e88d86194c2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0d13a9f0812f4ec58341619045702650 at {username='oryx', principal='oryx/127.18.203.252@KRBTEST.COM'} at 127.0.0.1:58658: session id = 0d13a9f0812f4ec58341619045702650-00000000000000000000000000000000
I20260201 03:03:11.415836 26028 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2: Tablet Copy: opened 0 blocks and 1 log segments
I20260201 03:03:11.417153 26593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260201 03:03:11.417467 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.417172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.0.0.1:33730)
0201 03:03:11.417310 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.417382 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.252:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260201 03:03:11.417740 26008 master.cc:679] Network error: unable to get registration information for peer 0d13a9f0812f4ec58341619045702650: Client connection negotiation failed: client connection to 127.18.203.252:35217: connect: Connection refused (error 111)
I20260201 03:03:11.419220 26593 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: tablet copy: Starting download of 0 data blocks...
I20260201 03:03:11.419380 26593 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: tablet copy: Starting download of 1 WAL segments...
I20260201 03:03:11.420970 26593 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260201 03:03:11.422489 26028 tablet_copy_service.cc:342] P 4cc37f0a8a064fa388608e88d86194c2: Request end of tablet copy session 0d13a9f0812f4ec58341619045702650-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.203.252@KRBTEST.COM'} at 127.0.0.1:58658
I20260201 03:03:11.422585 26028 tablet_copy_service.cc:434] P 4cc37f0a8a064fa388608e88d86194c2: ending tablet copy session 0d13a9f0812f4ec58341619045702650-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0d13a9f0812f4ec58341619045702650
W20260201 03:03:11.426771 26593 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260201 03:03:11.428964 26593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:11.430663 26780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:11.430691 26777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:11.430845 26778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:11.431064 26593 server_base.cc:1060] running on GCE node
I20260201 03:03:11.431205 26593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:11.431488 26593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:11.432621 26593 hybrid_clock.cc:648] HybridClock initialized: now 1769914991432616 us; error 41 us; skew 500 ppm
Feb 01 03:03:11 dist-test-slave-gf59 krb5kdc[25960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914991, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:11.434384 26593 init.cc:377] Logged in from keytab as oryx/127.18.203.252@KRBTEST.COM (short username oryx)
I20260201 03:03:11.434929 26593 webserver.cc:492] Webserver started at http://127.18.203.252:34303/ using document root <none> and password file <none>
I20260201 03:03:11.435127 26593 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:11.435170 26593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:11.436614 26593 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:11.437683 26787 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:11.437985 26593 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:11.438062 26593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/wal
uuid: "0d13a9f0812f4ec58341619045702650"
format_stamp: "Formatted at 2026-02-01 03:03:11 on dist-test-slave-gf59"
server_key: "9b43439e4777084c0af46e8ce7966eb5"
server_key_iv: "6d4110b0ea4d46e89507977c37b8c526"
server_key_version: "encryptionkey@0"
I20260201 03:03:11.438143 26593 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:11.452046 26593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:11.454068 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.453766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.0.0.1:33736)
0201 03:03:11.453910 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.453970 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.252:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260201 03:03:11.454315 26008 master.cc:679] Network error: unable to get registration information for peer 0d13a9f0812f4ec58341619045702650: Client connection negotiation failed: client connection to 127.18.203.252:35217: connect: Connection refused (error 111)
I20260201 03:03:11.454782 26593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:11.459300 26593 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.252:35217
I20260201 03:03:11.459441 26839 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.252:35217 every 8 connection(s)
I20260201 03:03:11.459939 26593 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/master-2/data/info.pb
I20260201 03:03:11.461313 26840 sys_catalog.cc:263] Verifying existing consensus state
I20260201 03:03:11.462066 26840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: Bootstrap starting.
I20260201 03:03:11.470992 26840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260201 03:03:11.471300 26840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: Bootstrap complete.
I20260201 03:03:11.471887 26840 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } }
I20260201 03:03:11.472009 26840 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d13a9f0812f4ec58341619045702650, State: Initialized, Role: LEARNER
I20260201 03:03:11.472110 26840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } }
I20260201 03:03:11.472440 26843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:11.472548 26843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: This master's current role is: LEARNER
I20260201 03:03:11.472596 26840 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:11.475432 26854 catalog_manager.cc:1269] Loaded cluster ID: ab56833fde3947a39e0b6397cf47f13e
I20260201 03:03:11.475488 26854 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: loading cluster ID for follower catalog manager: success
I20260201 03:03:11.477384 26854 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: acquiring CA information for follower catalog manager: success
I20260201 03:03:11.478600 26854 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260201 03:03:11.500406 26337 heartbeater.cc:499] Master 127.18.203.254:40003 was elected leader, sending a full tablet report...
I20260201 03:03:11.524919 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.522149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.0.0.1:33742)
0201 03:03:11.522317 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:11.522333 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:11.522421 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:11.522760 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:11.522763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:11.522785 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:11.523024 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.523032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.523935 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.523938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:11.524554 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:11.524562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.524657 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.524677 (+ 20us) client_negotiation.cc:769] Sending connection context
0201 03:03:11.524710 (+ 33us) client_negotiation.cc:240] Negotiation successful
0201 03:03:11.524765 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260201 03:03:11.525346 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:11.522297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.18.203.252:35217)
0201 03:03:11.522536 (+ 239us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:11.522539 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:11.522552 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:11.522594 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:11.522596 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:11.522644 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:11.522728 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:11.523137 (+ 409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.523838 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:11.524649 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:11.524959 (+ 310us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:11.525124 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:11.525171 (+ 47us) server_negotiation.cc:300] Negotiation successful
0201 03:03:11.525221 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260201 03:03:11.727267 26473 heartbeater.cc:499] Master 127.18.203.254:40003 was elected leader, sending a full tablet report...
I20260201 03:03:11.780242 26808 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260201 03:03:11.780717 26843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cc37f0a8a064fa388608e88d86194c2. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:11.780874 26843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: This master's current role is: LEARNER
I20260201 03:03:12.189944 26856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: NON_VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: true } } }
I20260201 03:03:12.190114 26856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:12.190991 26580 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2: attempting to promote NON_VOTER 0d13a9f0812f4ec58341619045702650 to VOTER
I20260201 03:03:12.191381 26580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } }
I20260201 03:03:12.192620 26808 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 1 LEARNER]: Refusing update from remote peer 4cc37f0a8a064fa388608e88d86194c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260201 03:03:12.192692 26631 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 FOLLOWER]: Refusing update from remote peer 4cc37f0a8a064fa388608e88d86194c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260201 03:03:12.193094 26580 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260201 03:03:12.193336 26856 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260201 03:03:12.195067 26858 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0d13a9f0812f4ec58341619045702650 (127.18.203.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } } }
I20260201 03:03:12.195492 26631 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0d13a9f0812f4ec58341619045702650 (127.18.203.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } } }
I20260201 03:03:12.196655 26858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } } }
I20260201 03:03:12.196758 26858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:12.196512 26808 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0d13a9f0812f4ec58341619045702650 (127.18.203.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } } }
I20260201 03:03:12.196820 26856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } } }
I20260201 03:03:12.196866 26858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } } }
I20260201 03:03:12.196905 26856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:12.196930 26858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:12.197291 26860 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } } }
I20260201 03:03:12.197386 26860 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: This master's current role is: FOLLOWER
I20260201 03:03:12.198161 26843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cc37f0a8a064fa388608e88d86194c2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } } }
I20260201 03:03:12.198272 26843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: This master's current role is: FOLLOWER
I20260201 03:03:12.445204 26862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.436164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.18.203.253:38411)
0201 03:03:12.436395 (+ 231us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.436398 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.436428 (+ 30us) server_negotiation.cc:408] Connection header received
0201 03:03:12.436542 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.436544 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.436587 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.436660 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:12.437444 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.437940 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.438521 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.438644 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.442502 (+ 3858us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:12.442519 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:12.442521 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:12.442546 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:12.443937 (+ 1391us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.444276 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.444278 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.444279 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.444317 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.444456 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.444458 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.444459 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.444598 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:12.444689 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.444960 (+ 271us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.445057 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260201 03:03:12.448314 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.446039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:58670)
0201 03:03:12.446278 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:12.446290 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:12.446359 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:12.446562 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:12.446564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:12.446576 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:12.446742 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.446746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.447450 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.447453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:12.448029 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.448035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.448120 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.448135 (+ 15us) client_negotiation.cc:769] Sending connection context
0201 03:03:12.448168 (+ 33us) client_negotiation.cc:240] Negotiation successful
0201 03:03:12.448209 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":89,"threads_started":1}
I20260201 03:03:12.448792 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.446099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.18.203.254:40003)
0201 03:03:12.446322 (+ 223us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.446325 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.446337 (+ 12us) server_negotiation.cc:408] Connection header received
0201 03:03:12.446436 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.446439 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.446480 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.446547 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:12.446814 (+ 267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.447371 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.448145 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.448567 (+ 422us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.448605 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.448646 (+ 41us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.448693 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260201 03:03:12.451887 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.449875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.253:38411 (local address 127.0.0.1:54730)
0201 03:03:12.449968 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:12.449979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:12.450039 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:12.450220 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:12.450222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:12.450236 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:12.450378 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.450382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.451000 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.451003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:12.451568 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.451575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.451705 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.451719 (+ 14us) client_negotiation.cc:769] Sending connection context
0201 03:03:12.451756 (+ 37us) client_negotiation.cc:240] Negotiation successful
0201 03:03:12.451801 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260201 03:03:12.452227 26862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.449974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54730 (local address 127.18.203.253:38411)
0201 03:03:12.450071 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.450075 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.450085 (+ 10us) server_negotiation.cc:408] Connection header received
0201 03:03:12.450114 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.450116 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.450150 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.450204 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:12.450446 (+ 242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.450888 (+ 442us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.451701 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.452022 (+ 321us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.452064 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.452105 (+ 41us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.452150 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260201 03:03:12.456513 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.453335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.0.0.1:33754)
0201 03:03:12.453438 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:12.453452 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:12.453529 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:12.454012 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:12.454015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:12.454055 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:12.454217 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.454222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.455281 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.455285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:12.456167 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.456178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.456278 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.456294 (+ 16us) client_negotiation.cc:769] Sending connection context
0201 03:03:12.456340 (+ 46us) client_negotiation.cc:240] Negotiation successful
0201 03:03:12.456394 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260201 03:03:12.457139 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.453460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.18.203.252:35217)
0201 03:03:12.453791 (+ 331us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.453796 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.453809 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:12.453860 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.453863 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.453912 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.453998 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:12.454336 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.455165 (+ 829us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.456315 (+ 1150us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.456845 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.456885 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.456942 (+ 57us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.457005 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":159,"threads_started":1}
I20260201 03:03:12.470499 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.459288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.18.203.252:35217)
0201 03:03:12.459743 (+ 455us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.459747 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.459759 (+ 12us) server_negotiation.cc:408] Connection header received
0201 03:03:12.459793 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.459797 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.459839 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.459908 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:12.461056 (+ 1148us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.461687 (+ 631us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.462496 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.462671 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.466725 (+ 4054us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:12.466769 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:12.466773 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:12.466803 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:12.468836 (+ 2033us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.469292 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.469296 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.469298 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.469348 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.469634 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.469637 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.469638 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.469843 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:12.469998 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.470203 (+ 205us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.470343 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370}
I20260201 03:03:12.474262 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.471359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:58682)
0201 03:03:12.471650 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:12.471665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:12.471765 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:12.472079 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:12.472083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:12.472102 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:12.472356 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.472365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.473121 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.473125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:12.473845 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.473855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.473962 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.473983 (+ 21us) client_negotiation.cc:769] Sending connection context
0201 03:03:12.474030 (+ 47us) client_negotiation.cc:240] Negotiation successful
0201 03:03:12.474095 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260201 03:03:12.474757 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.471509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.18.203.254:40003)
0201 03:03:12.471700 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.471705 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.471722 (+ 17us) server_negotiation.cc:408] Connection header received
0201 03:03:12.471922 (+ 200us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.471926 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.471979 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.472069 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:12.472473 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.472993 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.473969 (+ 976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.474537 (+ 568us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.474577 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.474620 (+ 43us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.474668 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260201 03:03:12.479128 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.476140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.253:38411 (local address 127.0.0.1:54734)
0201 03:03:12.476373 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:12.476387 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:12.476506 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:12.476726 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:12.476729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:12.476748 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:12.476934 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.476941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.477737 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.477740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:12.478552 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.478563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.478672 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.478690 (+ 18us) client_negotiation.cc:769] Sending connection context
0201 03:03:12.478938 (+ 248us) client_negotiation.cc:240] Negotiation successful
0201 03:03:12.479003 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260201 03:03:12.479398 26862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.476235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.18.203.253:38411)
0201 03:03:12.476388 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.476392 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.476440 (+ 48us) server_negotiation.cc:408] Connection header received
0201 03:03:12.476596 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.476600 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.476650 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.476711 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:12.477043 (+ 332us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.477638 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.478672 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.479147 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.479187 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.479242 (+ 55us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.479308 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260201 03:03:12.486271 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.480834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.0.0.1:33776)
0201 03:03:12.483690 (+ 2856us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:12.483706 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:12.483811 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:12.484011 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:12.484015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:12.484033 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:12.484269 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.484286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.484951 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.484955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:12.485851 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.485862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.485988 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.486010 (+ 22us) client_negotiation.cc:769] Sending connection context
0201 03:03:12.486062 (+ 52us) client_negotiation.cc:240] Negotiation successful
0201 03:03:12.486130 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2723,"spinlock_wait_cycles":896}
I20260201 03:03:12.486532 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.480886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.18.203.252:35217)
0201 03:03:12.481004 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.481008 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.483769 (+ 2761us) server_negotiation.cc:408] Connection header received
0201 03:03:12.483892 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.483894 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.483934 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.483998 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:12.484388 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.484854 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.485961 (+ 1107us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.486283 (+ 322us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.486326 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.486373 (+ 47us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.486426 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30,"spinlock_wait_cycles":3456}
I20260201 03:03:12.488145 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 26072
I20260201 03:03:12.494769 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:42487
--local_ip_for_outbound_sockets=127.18.203.193
--tserver_master_addrs=127.18.203.254:40003
--webserver_port=41329
--webserver_interface=127.18.203.193
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.203.254:40003,127.18.203.253:38411
--tserver_master_addrs=127.18.203.254:40003,127.18.203.253:38411,127.18.203.252:35217 with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:12.607312 26867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:12.607578 26867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:12.607681 26867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:12.610992 26867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:12.611073 26867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:12.611155 26867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:03:12.615573 26867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:42487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=41329
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:40003,127.18.203.253:38411,127.18.203.252:35217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:12.616736 26867 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:12.617594 26867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:12.624644 26873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:12.624589 26872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:12.624701 26867 server_base.cc:1060] running on GCE node
W20260201 03:03:12.624593 26875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:12.625196 26867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:12.625798 26867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:12.626941 26867 hybrid_clock.cc:648] HybridClock initialized: now 1769914992626925 us; error 38 us; skew 500 ppm
Feb 01 03:03:12 dist-test-slave-gf59 krb5kdc[25960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914992, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:12.630020 26867 init.cc:377] Logged in from keytab as oryx/127.18.203.193@KRBTEST.COM (short username oryx)
I20260201 03:03:12.631183 26867 webserver.cc:492] Webserver started at http://127.18.203.193:41329/ using document root <none> and password file <none>
I20260201 03:03:12.631807 26867 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:12.631884 26867 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:12.635555 26867 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:03:12.637670 26882 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:12.638717 26867 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:12.638820 26867 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "90f106a46c90471c960543c8320a0242"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "6f65505b129bed208eb3194b2661fa2c"
server_key_iv: "fd0e3606a2b363b24e9e06ca7299b26e"
server_key_version: "encryptionkey@0"
I20260201 03:03:12.639289 26867 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:12.662194 26867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:12.665187 26867 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:12.665390 26867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:12.665977 26867 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:12.666939 26867 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:12.666999 26867 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:12.667053 26867 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:12.667083 26867 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:12.676393 26867 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:42487
I20260201 03:03:12.676518 26995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:42487 every 8 connection(s)
I20260201 03:03:12.677440 26867 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:03:12.682132 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 26867
I20260201 03:03:12.682344 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 26208
Feb 01 03:03:12 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914992, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.253@KRBTEST.COM
Feb 01 03:03:12 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914992, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.252@KRBTEST.COM
Feb 01 03:03:12 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914992, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:12.690189 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:45423
--local_ip_for_outbound_sockets=127.18.203.194
--tserver_master_addrs=127.18.203.254:40003
--webserver_port=38785
--webserver_interface=127.18.203.194
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.203.254:40003,127.18.203.253:38411
--tserver_master_addrs=127.18.203.254:40003,127.18.203.253:38411,127.18.203.252:35217 with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260201 03:03:12.693194 26862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.680448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:40235 (local address 127.18.203.253:38411)
0201 03:03:12.680578 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.680583 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.680599 (+ 16us) server_negotiation.cc:408] Connection header received
0201 03:03:12.681038 (+ 439us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.681042 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.681094 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.681165 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:12.683479 (+ 2314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.684309 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.685249 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.685434 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.688785 (+ 3351us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:12.688807 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:12.688810 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:12.688841 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:12.691442 (+ 2601us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.691889 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.691893 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.691895 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.691948 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.692203 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.692207 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.692209 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.692363 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:12.692416 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.692899 (+ 483us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.693036 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260201 03:03:12.693198 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.680966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:34135 (local address 127.18.203.252:35217)
0201 03:03:12.681135 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.681139 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.681153 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:03:12.681193 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.681196 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.681244 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.681335 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:12.684269 (+ 2934us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.684972 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.685802 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.685976 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.689176 (+ 3200us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:12.689198 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:12.689201 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:12.689234 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:12.690891 (+ 1657us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.691594 (+ 703us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.691598 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.691600 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.691676 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.692032 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.692035 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.692036 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.692215 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:12.692285 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.692909 (+ 624us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.693040 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260201 03:03:12.693198 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.679173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:59097 (local address 127.18.203.254:40003)
0201 03:03:12.679316 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.679320 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.680021 (+ 701us) server_negotiation.cc:408] Connection header received
0201 03:03:12.681523 (+ 1502us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.681527 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.681581 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.681663 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:12.683478 (+ 1815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.684315 (+ 837us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.685238 (+ 923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.685434 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.689471 (+ 4037us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:12.689492 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:12.689495 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:12.689525 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:12.691028 (+ 1503us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.691594 (+ 566us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.691598 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.691600 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.691676 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.692022 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.692026 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.692027 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.692215 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:12.692287 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.692909 (+ 622us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.693042 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260201 03:03:12.694339 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.679434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.193:59097)
0201 03:03:12.679881 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:12.679918 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:12.680803 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:12.681688 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:12.681692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:12.681974 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:12.683304 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.683321 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.684465 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.684469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:12.685107 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.685112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.685671 (+ 559us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.686530 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:12.686573 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:12.689326 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:12.691135 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:12.691138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:12.691140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:12.691479 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:12.691785 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:12.691788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:12.691790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:12.691918 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:12.692368 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:12.692376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:12.692650 (+ 274us) client_negotiation.cc:769] Sending connection context
0201 03:03:12.693395 (+ 745us) client_negotiation.cc:240] Negotiation successful
0201 03:03:12.693530 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"spinlock_wait_cycles":66688,"thread_start_us":132,"threads_started":1}
I20260201 03:03:12.694358 27002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.680629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.18.203.193:34135)
0201 03:03:12.680856 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:12.680870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:12.680951 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:12.681358 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:12.681367 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:12.682361 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:12.684107 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.684116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.685114 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.685117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:12.685671 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.685679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.686086 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.686528 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:12.686581 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:12.689033 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:12.691038 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:12.691044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:12.691058 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:12.691479 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:12.691785 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:12.691788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:12.691790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:12.691918 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:12.692368 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:12.692376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:12.692650 (+ 274us) client_negotiation.cc:769] Sending connection context
0201 03:03:12.693395 (+ 745us) client_negotiation.cc:240] Negotiation successful
0201 03:03:12.693603 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":87,"spinlock_wait_cycles":28544,"thread_start_us":65,"threads_started":1}
I20260201 03:03:12.694339 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.680304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.253:38411 (local address 127.18.203.193:40235)
0201 03:03:12.680528 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:12.680541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:12.680810 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:12.681358 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:12.681368 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:12.681974 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:12.683319 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.683326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.684465 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.684469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:12.685103 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.685111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.685671 (+ 560us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.686535 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:12.686571 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:12.688616 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:12.691571 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:12.691574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:12.691576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:12.691781 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:12.692056 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:12.692058 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:12.692060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:12.692112 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:12.692471 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:12.692473 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:12.692648 (+ 175us) client_negotiation.cc:769] Sending connection context
0201 03:03:12.692842 (+ 194us) client_negotiation.cc:240] Negotiation successful
0201 03:03:12.693120 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"spinlock_wait_cycles":23296,"thread_start_us":78,"threads_started":1}
I20260201 03:03:12.695516 26997 heartbeater.cc:344] Connected to a master server at 127.18.203.254:40003
I20260201 03:03:12.695595 26996 heartbeater.cc:344] Connected to a master server at 127.18.203.252:35217
I20260201 03:03:12.695622 26998 heartbeater.cc:344] Connected to a master server at 127.18.203.253:38411
I20260201 03:03:12.695799 26997 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:12.695799 26998 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:12.695799 26996 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:12.696457 26996 heartbeater.cc:507] Master 127.18.203.252:35217 requested a full tablet report, sending...
I20260201 03:03:12.696439 26998 heartbeater.cc:507] Master 127.18.203.253:38411 requested a full tablet report, sending...
I20260201 03:03:12.696444 26997 heartbeater.cc:507] Master 127.18.203.254:40003 requested a full tablet report, sending...
I20260201 03:03:12.697971 26008 ts_manager.cc:194] Re-registered known tserver with Master: 90f106a46c90471c960543c8320a0242 (127.18.203.193:42487)
I20260201 03:03:12.698058 26804 ts_manager.cc:194] Registered new tserver with Master: 90f106a46c90471c960543c8320a0242 (127.18.203.193:42487)
I20260201 03:03:12.698082 26621 ts_manager.cc:194] Registered new tserver with Master: 90f106a46c90471c960543c8320a0242 (127.18.203.193:42487)
I20260201 03:03:12.698720 26008 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:59097
W20260201 03:03:12.805491 27003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:12.805732 27003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:12.805799 27003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:12.809303 27003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:12.809381 27003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:12.809466 27003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:03:12.813719 27003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:45423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=38785
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:40003,127.18.203.253:38411,127.18.203.252:35217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:12.814882 27003 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:12.815791 27003 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:12.822346 27009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:12.822417 27012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:12.822518 27003 server_base.cc:1060] running on GCE node
W20260201 03:03:12.822741 27010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:12.823213 27003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:12.823844 27003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:12.825008 27003 hybrid_clock.cc:648] HybridClock initialized: now 1769914992824997 us; error 54 us; skew 500 ppm
Feb 01 03:03:12 dist-test-slave-gf59 krb5kdc[25960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914992, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:12.827703 27003 init.cc:377] Logged in from keytab as oryx/127.18.203.194@KRBTEST.COM (short username oryx)
I20260201 03:03:12.828796 27003 webserver.cc:492] Webserver started at http://127.18.203.194:38785/ using document root <none> and password file <none>
I20260201 03:03:12.829388 27003 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:12.829456 27003 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:12.833213 27003 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260201 03:03:12.835211 27019 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:12.836300 27003 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:12.836410 27003 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "94bfc37fa718472db25e38cd9ad2366f"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "79579f45058f1e63b1888ca3ddbbbae3"
server_key_iv: "07b1c9f6f3d73492b1d79cc576fcf7ea"
server_key_version: "encryptionkey@0"
I20260201 03:03:12.836881 27003 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:12.865447 27003 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:12.868257 27003 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:12.868438 27003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:12.868996 27003 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:12.869946 27003 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:12.869997 27003 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:12.870059 27003 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:12.870082 27003 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:12.879361 27003 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:45423
I20260201 03:03:12.879385 27132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:45423 every 8 connection(s)
I20260201 03:03:12.880457 27003 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
Feb 01 03:03:12 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914992, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.253@KRBTEST.COM
Feb 01 03:03:12 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914992, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.252@KRBTEST.COM
I20260201 03:03:12.890067 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 27003
Feb 01 03:03:12 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914992, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:12.890260 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 26344
I20260201 03:03:12.895088 26862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.883481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:46955 (local address 127.18.203.253:38411)
0201 03:03:12.883600 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.883604 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.883641 (+ 37us) server_negotiation.cc:408] Connection header received
0201 03:03:12.884076 (+ 435us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.884079 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.884135 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.884189 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:12.885657 (+ 1468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.886142 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.886927 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.887128 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.890024 (+ 2896us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:12.890047 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:12.890050 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:12.890086 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:12.892802 (+ 2716us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.893390 (+ 588us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.893394 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.893397 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.893447 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.893769 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.893773 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.893775 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.894005 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:12.894075 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.894777 (+ 702us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.894910 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260201 03:03:12.895098 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.882191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:34563 (local address 127.18.203.254:40003)
0201 03:03:12.882345 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.882349 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.883011 (+ 662us) server_negotiation.cc:408] Connection header received
0201 03:03:12.884068 (+ 1057us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.884071 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.884128 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.884180 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:12.885548 (+ 1368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.886346 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.887056 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.887246 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.890710 (+ 3464us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:12.890732 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:12.890735 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:12.890769 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:12.892655 (+ 1886us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.893398 (+ 743us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.893402 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.893404 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.893468 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.893782 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.893786 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.893788 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.893988 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:12.894099 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.894778 (+ 679us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.894921 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260201 03:03:12.896327 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.883823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:47319 (local address 127.18.203.252:35217)
0201 03:03:12.883947 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:12.883951 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:12.884001 (+ 50us) server_negotiation.cc:408] Connection header received
0201 03:03:12.884046 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:12.884049 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:12.884100 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:12.884179 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:12.885607 (+ 1428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.886346 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.887054 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.887232 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.891218 (+ 3986us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:12.891241 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:12.891244 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:12.891288 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:12.894275 (+ 2987us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.894695 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.894699 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.894701 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.894754 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:12.895003 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:12.895006 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:12.895008 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:12.895190 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:12.895272 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:12.896060 (+ 788us) server_negotiation.cc:300] Negotiation successful
0201 03:03:12.896182 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260201 03:03:12.896509 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.883292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.253:38411 (local address 127.18.203.194:46955)
0201 03:03:12.883519 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:12.883532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:12.883850 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:12.884371 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:12.884379 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:12.885286 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:12.885524 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.885531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.886266 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.886270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:12.886792 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.886801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.887345 (+ 544us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.888155 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:12.888168 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:12.889840 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:12.892915 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:12.892918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:12.892921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:12.893270 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:12.893544 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:12.893546 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:12.893549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:12.893665 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:12.894154 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:12.894161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:12.894480 (+ 319us) client_negotiation.cc:769] Sending connection context
0201 03:03:12.895315 (+ 835us) client_negotiation.cc:240] Negotiation successful
0201 03:03:12.895703 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":119,"spinlock_wait_cycles":19200,"thread_start_us":66,"threads_started":1}
I20260201 03:03:12.896521 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.882452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.194:34563)
0201 03:03:12.882864 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:12.882900 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:12.883831 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:12.884370 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:12.884379 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:12.884824 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:12.885401 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.885412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.886498 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.886500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:12.886915 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.886925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.887368 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.888556 (+ 1188us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:12.888575 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:12.890557 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:12.892802 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:12.892809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:12.892822 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:12.893272 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:12.893588 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:12.893591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:12.893593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:12.893674 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:12.894171 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:12.894174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:12.894485 (+ 311us) client_negotiation.cc:769] Sending connection context
0201 03:03:12.895279 (+ 794us) client_negotiation.cc:240] Negotiation successful
0201 03:03:12.895596 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"spinlock_wait_cycles":5376,"thread_start_us":99,"threads_started":1}
I20260201 03:03:12.897444 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:12.883629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.18.203.194:47319)
0201 03:03:12.883856 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:12.883870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:12.883951 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:12.884371 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:12.884380 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:12.884839 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:12.885428 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.885436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.886449 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:12.886453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:12.886936 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:12.886944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:12.887345 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:12.888084 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:12.888115 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:12.890287 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:12.894387 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:12.894390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:12.894391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:12.894576 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:12.894857 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:12.894860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:12.894862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:12.894909 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:12.895801 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:12.895804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:12.895881 (+ 77us) client_negotiation.cc:769] Sending connection context
0201 03:03:12.896948 (+ 1067us) client_negotiation.cc:240] Negotiation successful
0201 03:03:12.897289 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"spinlock_wait_cycles":24192,"thread_start_us":68,"threads_started":1}
I20260201 03:03:12.897732 27135 heartbeater.cc:344] Connected to a master server at 127.18.203.253:38411
I20260201 03:03:12.897787 27134 heartbeater.cc:344] Connected to a master server at 127.18.203.254:40003
I20260201 03:03:12.897989 27134 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:12.897991 27135 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:12.898613 27134 heartbeater.cc:507] Master 127.18.203.254:40003 requested a full tablet report, sending...
I20260201 03:03:12.898612 27135 heartbeater.cc:507] Master 127.18.203.253:38411 requested a full tablet report, sending...
I20260201 03:03:12.898801 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:38661
--local_ip_for_outbound_sockets=127.18.203.195
--tserver_master_addrs=127.18.203.254:40003
--webserver_port=39745
--webserver_interface=127.18.203.195
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.203.254:40003,127.18.203.253:38411
--tserver_master_addrs=127.18.203.254:40003,127.18.203.253:38411,127.18.203.252:35217 with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260201 03:03:12.899078 27133 heartbeater.cc:344] Connected to a master server at 127.18.203.252:35217
I20260201 03:03:12.899140 27133 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:12.899294 27133 heartbeater.cc:507] Master 127.18.203.252:35217 requested a full tablet report, sending...
I20260201 03:03:12.899730 26621 ts_manager.cc:194] Registered new tserver with Master: 94bfc37fa718472db25e38cd9ad2366f (127.18.203.194:45423)
I20260201 03:03:12.899772 26008 ts_manager.cc:194] Re-registered known tserver with Master: 94bfc37fa718472db25e38cd9ad2366f (127.18.203.194:45423)
I20260201 03:03:12.900425 26008 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:34563
I20260201 03:03:12.900972 26804 ts_manager.cc:194] Registered new tserver with Master: 94bfc37fa718472db25e38cd9ad2366f (127.18.203.194:45423)
W20260201 03:03:13.009722 27140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:13.009989 27140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:13.010057 27140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:13.013363 27140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:13.013442 27140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:13.013530 27140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:03:13.017813 27140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:38661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=39745
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:40003,127.18.203.253:38411,127.18.203.252:35217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:13.018971 27140 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:13.019876 27140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:13.026742 27148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:13.026742 27145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:13.026767 27146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:13.027158 27140 server_base.cc:1060] running on GCE node
I20260201 03:03:13.027518 27140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:13.028091 27140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:13.029247 27140 hybrid_clock.cc:648] HybridClock initialized: now 1769914993029224 us; error 31 us; skew 500 ppm
Feb 01 03:03:13 dist-test-slave-gf59 krb5kdc[25960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914993, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:13.031891 27140 init.cc:377] Logged in from keytab as oryx/127.18.203.195@KRBTEST.COM (short username oryx)
I20260201 03:03:13.032969 27140 webserver.cc:492] Webserver started at http://127.18.203.195:39745/ using document root <none> and password file <none>
I20260201 03:03:13.033517 27140 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:13.033581 27140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:13.037302 27140 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260201 03:03:13.039155 27155 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:13.040241 27140 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:13.040338 27140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "e8a481e12029469aa778691177e5ab15"
format_stamp: "Formatted at 2026-02-01 03:03:10 on dist-test-slave-gf59"
server_key: "d998ce425b607a1096be5b20054ae944"
server_key_iv: "542ced3cb0774ee147657ade6266df60"
server_key_version: "encryptionkey@0"
I20260201 03:03:13.040751 27140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:13.070201 27140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:13.072970 27140 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:13.073164 27140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:13.073755 27140 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:13.074757 27140 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:13.074807 27140 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:13.074851 27140 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:13.074877 27140 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:13.083783 27140 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:38661
I20260201 03:03:13.083808 27268 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:38661 every 8 connection(s)
I20260201 03:03:13.084820 27140 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:03:13.086162 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 27140
Feb 01 03:03:13 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914993, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
Feb 01 03:03:13 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914993, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.252@KRBTEST.COM
Feb 01 03:03:13 dist-test-slave-gf59 krb5kdc[25960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914993, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.253@KRBTEST.COM
I20260201 03:03:13.097658 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.089297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.18.203.253:38411)
0201 03:03:13.089567 (+ 270us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.089572 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.089584 (+ 12us) server_negotiation.cc:408] Connection header received
0201 03:03:13.089619 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.089622 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.089664 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.090680 (+ 1016us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:13.090786 (+ 106us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.091548 (+ 762us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.092577 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.092775 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.093676 (+ 901us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:13.093697 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:13.093700 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:13.093728 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:13.095642 (+ 1914us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.096097 (+ 455us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.096101 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.096103 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.096152 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.096404 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.096408 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.096410 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.096588 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:13.096705 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.097358 (+ 653us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.097477 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260201 03:03:13.101739 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.088813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.18.203.254:40003)
0201 03:03:13.089162 (+ 349us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.089167 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.089180 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:13.089227 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.089240 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.089291 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.089382 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:13.090488 (+ 1106us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.091275 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.092346 (+ 1071us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.092532 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.094864 (+ 2332us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:13.094885 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:13.094889 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:13.094916 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:13.099928 (+ 5012us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.100378 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.100381 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.100383 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.100433 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.100691 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.100694 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.100696 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.100891 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:13.101352 (+ 461us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.101428 (+ 76us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.101559 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260201 03:03:13.101830 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.089843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:50881 (local address 127.18.203.252:35217)
0201 03:03:13.090114 (+ 271us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.090118 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.090132 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:03:13.090173 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.090176 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.090219 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.090291 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:13.093542 (+ 3251us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.094206 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.095229 (+ 1023us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.095427 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.097771 (+ 2344us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:13.097790 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:13.097793 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:13.097821 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:13.099855 (+ 2034us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.100246 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.100249 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.100250 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.100300 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.100530 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.100533 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.100535 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.100685 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:13.100763 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.101584 (+ 821us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.101689 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260201 03:03:13.102538 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.086945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.253:38411 (local address 127.18.203.195:53931)
0201 03:03:13.087427 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:13.087475 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:13.088522 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:13.089212 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:13.089224 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:13.089809 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:13.091010 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.091019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.091931 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.091934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:13.094730 (+ 2796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.094741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.095851 (+ 1110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.096229 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:13.096247 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:13.097600 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:13.099533 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:13.099537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:13.099539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:13.099763 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:13.100002 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:13.100005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:13.100007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:13.100138 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:13.100557 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:13.100565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:13.101016 (+ 451us) client_negotiation.cc:769] Sending connection context
0201 03:03:13.101296 (+ 280us) client_negotiation.cc:240] Negotiation successful
0201 03:03:13.101608 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"spinlock_wait_cycles":15360,"thread_start_us":113,"threads_started":1}
I20260201 03:03:13.102536 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.088353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.252:35217 (local address 127.18.203.195:50881)
0201 03:03:13.088583 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:13.088595 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:13.088676 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:13.093046 (+ 4370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:13.093049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:13.093228 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:13.093418 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.093425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.094351 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.094355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:13.094891 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.094901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.095000 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.095578 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:13.095593 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:13.097638 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:13.099958 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:13.099961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:13.099963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:13.100150 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:13.100390 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:13.100393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:13.100395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:13.100440 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:13.100801 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:13.100804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:13.101009 (+ 205us) client_negotiation.cc:769] Sending connection context
0201 03:03:13.101296 (+ 287us) client_negotiation.cc:240] Negotiation successful
0201 03:03:13.102063 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":458,"spinlock_wait_cycles":768,"thread_start_us":70,"threads_started":1}
I20260201 03:03:13.103231 26862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.086572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:53931 (local address 127.18.203.253:38411)
0201 03:03:13.086699 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.086703 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.087580 (+ 877us) server_negotiation.cc:408] Connection header received
0201 03:03:13.088767 (+ 1187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.088771 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.088829 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.088887 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:13.091141 (+ 2254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.091805 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.095496 (+ 3691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.095733 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.097737 (+ 2004us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:13.097757 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:13.097760 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:13.097788 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:13.099415 (+ 1627us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.099868 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.099871 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.099872 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.099915 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.100253 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.100256 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.100258 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.100423 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:13.100503 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.102963 (+ 2460us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.103091 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260201 03:03:13.103863 27269 heartbeater.cc:344] Connected to a master server at 127.18.203.252:35217
I20260201 03:03:13.103976 27271 heartbeater.cc:344] Connected to a master server at 127.18.203.253:38411
I20260201 03:03:13.104161 27269 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:13.104868 27269 heartbeater.cc:507] Master 127.18.203.252:35217 requested a full tablet report, sending...
I20260201 03:03:13.104161 27271 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:13.105618 27271 heartbeater.cc:507] Master 127.18.203.253:38411 requested a full tablet report, sending...
I20260201 03:03:13.106073 26804 ts_manager.cc:194] Registered new tserver with Master: e8a481e12029469aa778691177e5ab15 (127.18.203.195:38661)
I20260201 03:03:13.106303 26621 ts_manager.cc:194] Registered new tserver with Master: e8a481e12029469aa778691177e5ab15 (127.18.203.195:38661)
I20260201 03:03:13.105085 26008 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:03:13.108232 26008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:03:13.108899 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.089911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.18.203.252:35217)
0201 03:03:13.092997 (+ 3086us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.093002 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.097005 (+ 4003us) server_negotiation.cc:408] Connection header received
0201 03:03:13.097050 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.097054 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.097107 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.097188 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:13.098834 (+ 1646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.099590 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.100720 (+ 1130us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.100901 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.102217 (+ 1316us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:13.102240 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:13.102243 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:13.102268 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:13.107274 (+ 5006us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.107717 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.107721 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.107723 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.107772 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.108048 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.108051 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.108053 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.108234 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:13.108382 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.108597 (+ 215us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.108733 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3005,"thread_start_us":92,"threads_started":1}
I20260201 03:03:13.110602 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.088032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.195:58691)
0201 03:03:13.088268 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:13.088281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:13.088855 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:13.090392 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:13.090395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:13.090593 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:13.090912 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.090926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.091980 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.091995 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:13.092519 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.092528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.092766 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.093314 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:13.093351 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:13.095215 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:13.098832 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:13.098838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:13.098852 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:13.099193 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:13.104668 (+ 5475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:13.104671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:13.104673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:13.104727 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:13.110191 (+ 5464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:13.110194 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:13.110275 (+ 81us) client_negotiation.cc:769] Sending connection context
0201 03:03:13.110359 (+ 84us) client_negotiation.cc:240] Negotiation successful
0201 03:03:13.110472 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
I20260201 03:03:13.110707 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.088918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:58691 (local address 127.18.203.254:40003)
0201 03:03:13.089531 (+ 613us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.089535 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.089546 (+ 11us) server_negotiation.cc:408] Connection header received
0201 03:03:13.089577 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.089579 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.089617 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.089688 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:13.091081 (+ 1393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.091791 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.092647 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.092798 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.095377 (+ 2579us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:13.095396 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:13.095400 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:13.095428 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:13.097239 (+ 1811us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.104382 (+ 7143us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.104386 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.104388 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.104439 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.109883 (+ 5444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.109887 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.109889 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.110084 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:13.110200 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.110434 (+ 234us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.110565 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":79,"threads_started":1}
I20260201 03:03:13.111251 27270 heartbeater.cc:344] Connected to a master server at 127.18.203.254:40003
I20260201 03:03:13.111368 27270 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:13.111569 27270 heartbeater.cc:507] Master 127.18.203.254:40003 requested a full tablet report, sending...
I20260201 03:03:13.112212 26007 ts_manager.cc:194] Re-registered known tserver with Master: e8a481e12029469aa778691177e5ab15 (127.18.203.195:38661)
I20260201 03:03:13.112712 26007 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:58691
I20260201 03:03:13.127049 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.122122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:45423 (local address 127.0.0.1:38224)
0201 03:03:13.123087 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:13.123098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:13.123155 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:13.124241 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:13.124244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:13.124260 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:13.124448 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.124453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.125745 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.125749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:13.126630 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.126639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.126750 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.126774 (+ 24us) client_negotiation.cc:769] Sending connection context
0201 03:03:13.126825 (+ 51us) client_negotiation.cc:240] Negotiation successful
0201 03:03:13.126889 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":95,"threads_started":1}
I20260201 03:03:13.128286 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.122200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.18.203.193:42487)
0201 03:03:13.123396 (+ 1196us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.123565 (+ 169us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.123592 (+ 27us) server_negotiation.cc:408] Connection header received
0201 03:03:13.123718 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.123725 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.123927 (+ 202us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.124549 (+ 622us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:13.124681 (+ 132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.125794 (+ 1113us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.126965 (+ 1171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.127848 (+ 883us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.127987 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.128066 (+ 79us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.128145 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":96,"threads_started":1}
I20260201 03:03:13.128286 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.122506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.18.203.194:45423)
0201 03:03:13.123766 (+ 1260us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.123773 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.123796 (+ 23us) server_negotiation.cc:408] Connection header received
0201 03:03:13.123885 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.123891 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.124083 (+ 192us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.124568 (+ 485us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:13.124675 (+ 107us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.125627 (+ 952us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.127360 (+ 1733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.127913 (+ 553us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.127989 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.128056 (+ 67us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.128148 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1159,"thread_start_us":109,"threads_started":1}
I20260201 03:03:13.129470 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.123266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:38661 (local address 127.0.0.1:51284)
0201 03:03:13.124094 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:13.124108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:13.124196 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:13.126646 (+ 2450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:13.126650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:13.126668 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:13.126897 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.126903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.128196 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.128200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:13.129111 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.129121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.129222 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.129239 (+ 17us) client_negotiation.cc:769] Sending connection context
0201 03:03:13.129278 (+ 39us) client_negotiation.cc:240] Negotiation successful
0201 03:03:13.129336 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":108,"threads_started":1}
I20260201 03:03:13.130429 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.123725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51284 (local address 127.18.203.195:38661)
0201 03:03:13.126088 (+ 2363us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.126095 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.126117 (+ 22us) server_negotiation.cc:408] Connection header received
0201 03:03:13.126197 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.126202 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.126382 (+ 180us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.126515 (+ 133us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:13.127014 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.128073 (+ 1059us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.129677 (+ 1604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.130122 (+ 445us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.130211 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.130263 (+ 52us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.130327 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2248,"thread_start_us":130,"threads_started":1}
I20260201 03:03:13.131253 27067 tablet_service.cc:1505] Processing CreateTablet for tablet 4263d5d1e49f4a78924bfbdf410acd07 (DEFAULT_TABLE table=test-table [id=b88f38a17d7c464cb646b8837e13b616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:13.132216 27067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4263d5d1e49f4a78924bfbdf410acd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:13.132738 27203 tablet_service.cc:1505] Processing CreateTablet for tablet 4263d5d1e49f4a78924bfbdf410acd07 (DEFAULT_TABLE table=test-table [id=b88f38a17d7c464cb646b8837e13b616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:13.133522 27203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4263d5d1e49f4a78924bfbdf410acd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:13.135439 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.121769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:42487 (local address 127.0.0.1:40268)
0201 03:03:13.122796 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:13.122812 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:13.122927 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:13.124087 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:13.124091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:13.124111 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:13.124402 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.124409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.125897 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.125900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:13.126696 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.126704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.126782 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.126793 (+ 11us) client_negotiation.cc:769] Sending connection context
0201 03:03:13.126836 (+ 43us) client_negotiation.cc:240] Negotiation successful
0201 03:03:13.126889 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":127,"threads_started":1}
I20260201 03:03:13.138504 26930 tablet_service.cc:1505] Processing CreateTablet for tablet 4263d5d1e49f4a78924bfbdf410acd07 (DEFAULT_TABLE table=test-table [id=b88f38a17d7c464cb646b8837e13b616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:13.139200 27300 tablet_bootstrap.cc:492] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f: Bootstrap starting.
I20260201 03:03:13.139576 26930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4263d5d1e49f4a78924bfbdf410acd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:13.140301 27301 tablet_bootstrap.cc:492] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15: Bootstrap starting.
I20260201 03:03:13.142187 27300 tablet_bootstrap.cc:654] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:13.142709 27301 tablet_bootstrap.cc:654] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:13.143064 27300 log.cc:826] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:13.143479 27301 log.cc:826] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:13.145462 27301 tablet_bootstrap.cc:492] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15: No bootstrap required, opened a new log
I20260201 03:03:13.145463 27300 tablet_bootstrap.cc:492] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f: No bootstrap required, opened a new log
I20260201 03:03:13.145673 27300 ts_tablet_manager.cc:1403] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260201 03:03:13.145673 27301 ts_tablet_manager.cc:1403] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260201 03:03:13.146241 27304 tablet_bootstrap.cc:492] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242: Bootstrap starting.
I20260201 03:03:13.148538 27304 tablet_bootstrap.cc:654] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:13.148681 27301 raft_consensus.cc:359] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } } peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } }
I20260201 03:03:13.148682 27300 raft_consensus.cc:359] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } } peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } }
I20260201 03:03:13.148895 27300 raft_consensus.cc:385] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:13.148895 27301 raft_consensus.cc:385] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:13.148952 27300 raft_consensus.cc:740] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94bfc37fa718472db25e38cd9ad2366f, State: Initialized, Role: FOLLOWER
I20260201 03:03:13.148952 27301 raft_consensus.cc:740] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a481e12029469aa778691177e5ab15, State: Initialized, Role: FOLLOWER
I20260201 03:03:13.149343 27304 log.cc:826] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:13.149406 27300 consensus_queue.cc:260] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } } peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } }
I20260201 03:03:13.149406 27301 consensus_queue.cc:260] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } } peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } }
I20260201 03:03:13.150295 27301 ts_tablet_manager.cc:1434] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260201 03:03:13.150287 27300 ts_tablet_manager.cc:1434] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260201 03:03:13.150656 27270 heartbeater.cc:499] Master 127.18.203.254:40003 was elected leader, sending a full tablet report...
I20260201 03:03:13.151554 27134 heartbeater.cc:499] Master 127.18.203.254:40003 was elected leader, sending a full tablet report...
I20260201 03:03:13.152889 27308 raft_consensus.cc:493] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:03:13.153004 27308 raft_consensus.cc:515] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } } peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } }
I20260201 03:03:13.157809 27304 tablet_bootstrap.cc:492] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242: No bootstrap required, opened a new log
I20260201 03:03:13.158062 27304 ts_tablet_manager.cc:1403] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260201 03:03:13.158193 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.154934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:42487 (local address 127.18.203.194:44833)
0201 03:03:13.155082 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:13.155098 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:13.155181 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:13.155825 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:13.155829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:13.155846 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:13.156079 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.156089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.157035 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.157039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:13.157832 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.157842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.157944 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.157966 (+ 22us) client_negotiation.cc:769] Sending connection context
0201 03:03:13.158012 (+ 46us) client_negotiation.cc:240] Negotiation successful
0201 03:03:13.158078 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260201 03:03:13.159020 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.155934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:38661 (local address 127.18.203.194:50045)
0201 03:03:13.156164 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:13.156177 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:13.156249 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:13.156772 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:13.156775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:13.156792 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:13.157021 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.157028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.157973 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.157977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:13.158689 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:13.158697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.158797 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.158813 (+ 16us) client_negotiation.cc:769] Sending connection context
0201 03:03:13.158859 (+ 46us) client_negotiation.cc:240] Negotiation successful
0201 03:03:13.158913 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260201 03:03:13.159260 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.155366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:44833 (local address 127.18.203.193:42487)
0201 03:03:13.155493 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.155497 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.155510 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:13.155555 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.155558 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.155635 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.155706 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:13.156210 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.156916 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.158575 (+ 1659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.159015 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.159051 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.159098 (+ 47us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.159153 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260201 03:03:13.159523 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.156395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:50045 (local address 127.18.203.195:38661)
0201 03:03:13.156510 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.156515 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.156527 (+ 12us) server_negotiation.cc:408] Connection header received
0201 03:03:13.156565 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.156568 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.156607 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.156674 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:13.157161 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.157843 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.158807 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.159272 (+ 465us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.159317 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.159362 (+ 45us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.159418 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260201 03:03:13.160274 27308 leader_election.cc:290] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90f106a46c90471c960543c8320a0242 (127.18.203.193:42487), e8a481e12029469aa778691177e5ab15 (127.18.203.195:38661)
I20260201 03:03:13.160434 27223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4263d5d1e49f4a78924bfbdf410acd07" candidate_uuid: "94bfc37fa718472db25e38cd9ad2366f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a481e12029469aa778691177e5ab15" is_pre_election: true
I20260201 03:03:13.160413 26950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4263d5d1e49f4a78924bfbdf410acd07" candidate_uuid: "94bfc37fa718472db25e38cd9ad2366f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f106a46c90471c960543c8320a0242" is_pre_election: true
I20260201 03:03:13.160629 27223 raft_consensus.cc:2468] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94bfc37fa718472db25e38cd9ad2366f in term 0.
I20260201 03:03:13.161015 27022 leader_election.cc:304] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94bfc37fa718472db25e38cd9ad2366f, e8a481e12029469aa778691177e5ab15; no voters:
I20260201 03:03:13.161230 27308 raft_consensus.cc:2804] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:03:13.161302 27308 raft_consensus.cc:493] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:03:13.161350 27308 raft_consensus.cc:3060] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Advancing to term 1
W20260201 03:03:13.161329 27022 leader_election.cc:343] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90f106a46c90471c960543c8320a0242 (127.18.203.193:42487): Illegal state: must be running to vote when last-logged opid is not known
I20260201 03:03:13.161263 27304 raft_consensus.cc:359] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } } peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } }
I20260201 03:03:13.161437 27304 raft_consensus.cc:385] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:13.161495 27304 raft_consensus.cc:740] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f106a46c90471c960543c8320a0242, State: Initialized, Role: FOLLOWER
I20260201 03:03:13.161916 27304 consensus_queue.cc:260] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } } peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } }
I20260201 03:03:13.162343 27308 raft_consensus.cc:515] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } } peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } }
I20260201 03:03:13.162580 26997 heartbeater.cc:499] Master 127.18.203.254:40003 was elected leader, sending a full tablet report...
I20260201 03:03:13.162973 26950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4263d5d1e49f4a78924bfbdf410acd07" candidate_uuid: "94bfc37fa718472db25e38cd9ad2366f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f106a46c90471c960543c8320a0242"
I20260201 03:03:13.163102 26950 raft_consensus.cc:3060] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:13.163218 27308 leader_election.cc:290] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [CANDIDATE]: Term 1 election: Requested vote from peers 90f106a46c90471c960543c8320a0242 (127.18.203.193:42487), e8a481e12029469aa778691177e5ab15 (127.18.203.195:38661)
I20260201 03:03:13.164237 26950 raft_consensus.cc:2468] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94bfc37fa718472db25e38cd9ad2366f in term 1.
I20260201 03:03:13.164516 27304 ts_tablet_manager.cc:1434] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260201 03:03:13.164628 27223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4263d5d1e49f4a78924bfbdf410acd07" candidate_uuid: "94bfc37fa718472db25e38cd9ad2366f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a481e12029469aa778691177e5ab15"
I20260201 03:03:13.164746 27223 raft_consensus.cc:3060] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:13.165870 27223 raft_consensus.cc:2468] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94bfc37fa718472db25e38cd9ad2366f in term 1.
I20260201 03:03:13.166203 27022 leader_election.cc:304] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94bfc37fa718472db25e38cd9ad2366f, e8a481e12029469aa778691177e5ab15; no voters:
I20260201 03:03:13.166468 27308 raft_consensus.cc:2804] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:13.166566 27308 raft_consensus.cc:697] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 1 LEADER]: Becoming Leader. State: Replica: 94bfc37fa718472db25e38cd9ad2366f, State: Running, Role: LEADER
I20260201 03:03:13.166862 27308 consensus_queue.cc:237] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } } peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } }
I20260201 03:03:13.170058 26008 catalog_manager.cc:5654] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f reported cstate change: term changed from 0 to 1, leader changed from <none> to 94bfc37fa718472db25e38cd9ad2366f (127.18.203.194). New cstate: current_term: 1 leader_uuid: "94bfc37fa718472db25e38cd9ad2366f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } health_report { overall_health: UNKNOWN } } }
W20260201 03:03:13.178810 26999 tablet.cc:2378] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:03:13.181857 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.179209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.18.203.194:45423)
0201 03:03:13.179354 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.179358 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.179374 (+ 16us) server_negotiation.cc:408] Connection header received
0201 03:03:13.179441 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.179443 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.179495 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.179573 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:03:13.179877 (+ 304us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.180364 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.180926 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.181056 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.181138 (+ 82us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:03:13.181565 (+ 427us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:03:13.181655 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.181687 (+ 32us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.181729 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260201 03:03:13.187078 26950 raft_consensus.cc:1275] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242 [term 1 FOLLOWER]: Refusing update from remote peer 94bfc37fa718472db25e38cd9ad2366f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:13.187101 27223 raft_consensus.cc:1275] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15 [term 1 FOLLOWER]: Refusing update from remote peer 94bfc37fa718472db25e38cd9ad2366f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:13.187724 27308 consensus_queue.cc:1048] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:13.187944 27306 consensus_queue.cc:1048] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:13.196468 27312 mvcc.cc:204] Tried to move back new op lower bound from 7249571812087619584 to 7249571812014465024. Current Snapshot: MvccSnapshot[applied={T|T < 7249571812087619584}]
I20260201 03:03:13.203470 27313 mvcc.cc:204] Tried to move back new op lower bound from 7249571812087619584 to 7249571812014465024. Current Snapshot: MvccSnapshot[applied={T|T < 7249571812087619584 or (T in {7249571812087619584})}]
I20260201 03:03:13.204303 26007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:13.204438 26007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:13.209378 26007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 4cc37f0a8a064fa388608e88d86194c2: Sending DeleteTablet for 3 replicas of tablet 4263d5d1e49f4a78924bfbdf410acd07
I20260201 03:03:13.210070 27067 tablet_service.cc:1552] Processing DeleteTablet for tablet 4263d5d1e49f4a78924bfbdf410acd07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:13 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:38224
I20260201 03:03:13.210059 26930 tablet_service.cc:1552] Processing DeleteTablet for tablet 4263d5d1e49f4a78924bfbdf410acd07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:13 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:40268
I20260201 03:03:13.210232 27203 tablet_service.cc:1552] Processing DeleteTablet for tablet 4263d5d1e49f4a78924bfbdf410acd07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:13 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:51284
I20260201 03:03:13.210561 27322 tablet_replica.cc:333] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f: stopping tablet replica
I20260201 03:03:13.210824 27322 raft_consensus.cc:2243] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:03:13.211239 27322 raft_consensus.cc:2272] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:13.211604 27323 tablet_replica.cc:333] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15: stopping tablet replica
I20260201 03:03:13.211880 27323 raft_consensus.cc:2243] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:13.212210 27323 raft_consensus.cc:2272] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:13.212795 27322 ts_tablet_manager.cc:1916] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:13.212817 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 25976
I20260201 03:03:13.214555 27323 ts_tablet_manager.cc:1916] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:13.215983 27322 ts_tablet_manager.cc:1929] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:13.216084 27322 log.cc:1199] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/4263d5d1e49f4a78924bfbdf410acd07
I20260201 03:03:13.216408 27322 ts_tablet_manager.cc:1950] T 4263d5d1e49f4a78924bfbdf410acd07 P 94bfc37fa718472db25e38cd9ad2366f: Deleting consensus metadata
I20260201 03:03:13.216933 27323 ts_tablet_manager.cc:1929] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:13.217023 27323 log.cc:1199] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/4263d5d1e49f4a78924bfbdf410acd07
I20260201 03:03:13.217329 27323 ts_tablet_manager.cc:1950] T 4263d5d1e49f4a78924bfbdf410acd07 P e8a481e12029469aa778691177e5ab15: Deleting consensus metadata
I20260201 03:03:13.219828 27321 tablet_replica.cc:333] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242: stopping tablet replica
I20260201 03:03:13.220088 27321 raft_consensus.cc:2243] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:13.220386 27321 raft_consensus.cc:2272] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260201 03:03:13.221338 27022 connection.cc:537] server connection from 127.0.0.1:38224 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260201 03:03:13.221616 27156 connection.cc:537] server connection from 127.0.0.1:51284 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260201 03:03:13.221973 27321 ts_tablet_manager.cc:1916] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:13.224016 27321 ts_tablet_manager.cc:1929] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:13.224113 27321 log.cc:1199] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-0/wal/wals/4263d5d1e49f4a78924bfbdf410acd07
I20260201 03:03:13.224393 27321 ts_tablet_manager.cc:1950] T 4263d5d1e49f4a78924bfbdf410acd07 P 90f106a46c90471c960543c8320a0242: Deleting consensus metadata
W20260201 03:03:13.225353 26886 connection.cc:441] server connection from 127.0.0.1:40268 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40268 (request call id 1) could send its response
I20260201 03:03:13.234663 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.226521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.18.203.252:35217)
0201 03:03:13.226663 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.226668 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.226820 (+ 152us) server_negotiation.cc:408] Connection header received
0201 03:03:13.226945 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.226949 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.226999 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.227049 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:13.228397 (+ 1348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.229052 (+ 655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.229723 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.230209 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.231081 (+ 872us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:13.231103 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:13.231106 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:13.231135 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:13.232964 (+ 1829us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.233414 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.233418 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.233420 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.233471 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.233706 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.233709 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.233711 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.233907 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:13.234015 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.234373 (+ 358us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.234488 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260201 03:03:13.234699 26862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:13.226265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.18.203.253:38411)
0201 03:03:13.226400 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:13.226405 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:13.226421 (+ 16us) server_negotiation.cc:408] Connection header received
0201 03:03:13.226743 (+ 322us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:13.226747 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:13.226808 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:13.226891 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:13.227965 (+ 1074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.228759 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:13.229755 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:13.229955 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:13.231424 (+ 1469us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:13.231444 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:13.231448 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:13.231480 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:13.233149 (+ 1669us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.233594 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.233598 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.233600 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.233652 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:13.233877 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:13.233880 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:13.233882 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:13.234042 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:13.234158 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:13.234409 (+ 251us) server_negotiation.cc:300] Negotiation successful
0201 03:03:13.234539 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260201 03:03:14.199227 27330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.198754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.194:58813)
0201 03:03:14.199009 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.199101 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
W20260201 03:03:14.199894 27134 heartbeater.cc:646] Failed to heartbeat to 127.18.203.254:40003 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
I20260201 03:03:14.200373 27331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.200000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.193:35657)
0201 03:03:14.200219 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.200283 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
W20260201 03:03:14.200589 26997 heartbeater.cc:646] Failed to heartbeat to 127.18.203.254:40003 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
I20260201 03:03:14.202939 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.202538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.195:36455)
0201 03:03:14.202747 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.202824 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
W20260201 03:03:14.203272 27270 heartbeater.cc:646] Failed to heartbeat to 127.18.203.254:40003 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
I20260201 03:03:14.709091 27333 raft_consensus.cc:493] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc37f0a8a064fa388608e88d86194c2)
I20260201 03:03:14.709263 27333 raft_consensus.cc:515] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } }
I20260201 03:03:14.709828 27333 leader_election.cc:290] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc37f0a8a064fa388608e88d86194c2 (127.18.203.254:40003), 0d13a9f0812f4ec58341619045702650 (127.18.203.252:35217)
I20260201 03:03:14.709982 27334 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc37f0a8a064fa388608e88d86194c2)
I20260201 03:03:14.710165 27334 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } }
I20260201 03:03:14.710485 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.710053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:59188)
0201 03:03:14.710289 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.710376 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
W20260201 03:03:14.710646 26608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
I20260201 03:03:14.710703 26808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d13a9f0812f4ec58341619045702650" is_pre_election: true
I20260201 03:03:14.710875 26808 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a7a00d1b784a3fa9cd98e967c8eed4 in term 1.
I20260201 03:03:14.711252 27334 leader_election.cc:290] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc37f0a8a064fa388608e88d86194c2 (127.18.203.254:40003), d3a7a00d1b784a3fa9cd98e967c8eed4 (127.18.203.253:38411)
I20260201 03:03:14.711402 26608 leader_election.cc:304] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d13a9f0812f4ec58341619045702650, d3a7a00d1b784a3fa9cd98e967c8eed4; no voters:
I20260201 03:03:14.711483 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.711248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:59196)
0201 03:03:14.711364 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.711416 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260201 03:03:14.711561 26608 leader_election.cc:336] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc37f0a8a064fa388608e88d86194c2 (127.18.203.254:40003): Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
I20260201 03:03:14.711588 27333 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260201 03:03:14.711683 27333 raft_consensus.cc:493] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc37f0a8a064fa388608e88d86194c2)
I20260201 03:03:14.711723 27333 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 1 FOLLOWER]: Advancing to term 2
I20260201 03:03:14.711902 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.711385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:59210)
0201 03:03:14.711748 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.711818 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":51,"threads_started":1}
I20260201 03:03:14.711952 26631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d13a9f0812f4ec58341619045702650" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" is_pre_election: true
W20260201 03:03:14.712057 26791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
I20260201 03:03:14.712849 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.712638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:59212)
0201 03:03:14.712754 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.712799 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260201 03:03:14.712971 26791 leader_election.cc:336] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc37f0a8a064fa388608e88d86194c2 (127.18.203.254:40003): Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
I20260201 03:03:14.713002 27333 raft_consensus.cc:515] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } }
I20260201 03:03:14.713248 26631 raft_consensus.cc:2393] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d13a9f0812f4ec58341619045702650 in current term 2: Already voted for candidate d3a7a00d1b784a3fa9cd98e967c8eed4 in this term.
I20260201 03:03:14.713332 27333 leader_election.cc:290] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc37f0a8a064fa388608e88d86194c2 (127.18.203.254:40003), 0d13a9f0812f4ec58341619045702650 (127.18.203.252:35217)
I20260201 03:03:14.713668 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.713460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:59226)
0201 03:03:14.713564 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.713606 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260201 03:03:14.713824 26808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d13a9f0812f4ec58341619045702650"
I20260201 03:03:14.713847 26791 leader_election.cc:304] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d13a9f0812f4ec58341619045702650; no voters: 4cc37f0a8a064fa388608e88d86194c2, d3a7a00d1b784a3fa9cd98e967c8eed4
I20260201 03:03:14.713922 26808 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 1 FOLLOWER]: Advancing to term 2
I20260201 03:03:14.714286 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.714051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:59228)
0201 03:03:14.714184 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.714230 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260201 03:03:14.714385 26608 leader_election.cc:336] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc37f0a8a064fa388608e88d86194c2 (127.18.203.254:40003): Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
I20260201 03:03:14.714897 26808 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a7a00d1b784a3fa9cd98e967c8eed4 in term 2.
I20260201 03:03:14.714998 27334 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260201 03:03:14.715168 26608 leader_election.cc:304] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d13a9f0812f4ec58341619045702650, d3a7a00d1b784a3fa9cd98e967c8eed4; no voters: 4cc37f0a8a064fa388608e88d86194c2
I20260201 03:03:14.715361 27333 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 2 FOLLOWER]: Leader election won for term 2
I20260201 03:03:14.715515 27333 raft_consensus.cc:697] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [term 2 LEADER]: Becoming Leader. State: Replica: d3a7a00d1b784a3fa9cd98e967c8eed4, State: Running, Role: LEADER
I20260201 03:03:14.715759 27333 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } }
I20260201 03:03:14.716738 27339 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3a7a00d1b784a3fa9cd98e967c8eed4. Latest consensus state: current_term: 2 leader_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } } }
I20260201 03:03:14.716846 27339 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:14.717139 27341 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:03:14.719182 27341 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:03:14.719564 27341 catalog_manager.cc:1269] Loaded cluster ID: ab56833fde3947a39e0b6397cf47f13e
I20260201 03:03:14.719640 27341 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:03:14.720188 27341 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:03:14.720569 27341 catalog_manager.cc:6038] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: Loaded TSK: 0
I20260201 03:03:14.721263 27341 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:03:14.752256 26620 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:03:14.754628 26620 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:03:14.757369 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.757152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:59242)
0201 03:03:14.757251 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.757301 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260201 03:03:14.757630 26808 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [term 2 FOLLOWER]: Refusing update from remote peer d3a7a00d1b784a3fa9cd98e967c8eed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260201 03:03:14.758059 27333 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260201 03:03:14.758363 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.758143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.0.0.1:59246)
0201 03:03:14.758251 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.758302 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260201 03:03:14.758512 26608 consensus_peers.cc:597] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 -> Peer 4cc37f0a8a064fa388608e88d86194c2 (127.18.203.254:40003): Couldn't send request to peer 4cc37f0a8a064fa388608e88d86194c2. Status: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260201 03:03:14.759414 27334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3a7a00d1b784a3fa9cd98e967c8eed4. Latest consensus state: current_term: 2 leader_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } } }
I20260201 03:03:14.759519 27334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: This master's current role is: FOLLOWER
I20260201 03:03:14.760573 27333 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } } }
I20260201 03:03:14.760685 27333 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:14.760823 27334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4cc37f0a8a064fa388608e88d86194c2" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 40003 } } peers { permanent_uuid: "d3a7a00d1b784a3fa9cd98e967c8eed4" member_type: VOTER last_known_addr { host: "127.18.203.253" port: 38411 } attrs { promote: false } } peers { permanent_uuid: "0d13a9f0812f4ec58341619045702650" member_type: VOTER last_known_addr { host: "127.18.203.252" port: 35217 } attrs { promote: false } } }
I20260201 03:03:14.761080 27343 mvcc.cc:204] Tried to move back new op lower bound from 7249571818522525696 to 7249571818358140928. Current Snapshot: MvccSnapshot[applied={T|T < 7249571818522525696}]
I20260201 03:03:14.761461 27334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d13a9f0812f4ec58341619045702650 [sys.catalog]: This master's current role is: FOLLOWER
I20260201 03:03:14.769308 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.765995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:45423 (local address 127.0.0.1:38250)
0201 03:03:14.766148 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.766165 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:14.766249 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:14.766855 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:14.766858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:14.766877 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:14.767090 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:14.767098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.768068 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:14.768072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:14.768923 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:14.768932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.769040 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:14.769063 (+ 23us) client_negotiation.cc:769] Sending connection context
0201 03:03:14.769111 (+ 48us) client_negotiation.cc:240] Negotiation successful
0201 03:03:14.769173 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260201 03:03:14.769675 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.766347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:38661 (local address 127.0.0.1:51300)
0201 03:03:14.767224 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.767237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:14.767334 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:14.767574 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:14.767578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:14.767596 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:14.767804 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:14.767810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.768636 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:14.768640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:14.769336 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:14.769344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.769441 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:14.769459 (+ 18us) client_negotiation.cc:769] Sending connection context
0201 03:03:14.769506 (+ 47us) client_negotiation.cc:240] Negotiation successful
0201 03:03:14.769566 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":84,"threads_started":1}
I20260201 03:03:14.770210 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.766643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:42487 (local address 127.0.0.1:40284)
0201 03:03:14.767502 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.767512 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:14.767574 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:14.767872 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:14.767875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:14.767893 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:14.768118 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:14.768125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.769084 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:14.769087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:14.769867 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:14.769877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.769974 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:14.769990 (+ 16us) client_negotiation.cc:769] Sending connection context
0201 03:03:14.770035 (+ 45us) client_negotiation.cc:240] Negotiation successful
0201 03:03:14.770098 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":85,"threads_started":1}
I20260201 03:03:14.770581 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.766893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.18.203.195:38661)
0201 03:03:14.767124 (+ 231us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:14.767129 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:14.767312 (+ 183us) server_negotiation.cc:408] Connection header received
0201 03:03:14.767416 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:14.767419 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:14.767470 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:14.767554 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:14.767930 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.768540 (+ 610us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:14.769837 (+ 1297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.770305 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:14.770344 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:14.770396 (+ 52us) server_negotiation.cc:300] Negotiation successful
0201 03:03:14.770458 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260201 03:03:14.770725 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.767205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.18.203.193:42487)
0201 03:03:14.767463 (+ 258us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:14.767467 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:14.767669 (+ 202us) server_negotiation.cc:408] Connection header received
0201 03:03:14.767725 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:14.767729 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:14.767776 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:14.767848 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:14.768230 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.768963 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:14.769989 (+ 1026us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.770468 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:14.770501 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:14.770547 (+ 46us) server_negotiation.cc:300] Negotiation successful
0201 03:03:14.770610 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":94,"threads_started":1}
I20260201 03:03:14.771083 27203 tablet_service.cc:1505] Processing CreateTablet for tablet f41b3f300eae4d268b97c510bba8d1c3 (DEFAULT_TABLE table=test-table [id=e1d5085ac12d4bb8824dd8b2836c6a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:14.771183 26930 tablet_service.cc:1505] Processing CreateTablet for tablet f41b3f300eae4d268b97c510bba8d1c3 (DEFAULT_TABLE table=test-table [id=e1d5085ac12d4bb8824dd8b2836c6a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:14.771385 27203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41b3f300eae4d268b97c510bba8d1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:14.771420 26930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41b3f300eae4d268b97c510bba8d1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:14.771919 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.766094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.18.203.194:45423)
0201 03:03:14.766589 (+ 495us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:14.766594 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:14.766609 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:03:14.766662 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:14.766665 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:14.766721 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:14.766846 (+ 125us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:14.767207 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.767967 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:14.769983 (+ 2016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.770613 (+ 630us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:14.770647 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:14.771717 (+ 1070us) server_negotiation.cc:300] Negotiation successful
0201 03:03:14.771798 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":74,"threads_started":1}
I20260201 03:03:14.772533 27067 tablet_service.cc:1505] Processing CreateTablet for tablet f41b3f300eae4d268b97c510bba8d1c3 (DEFAULT_TABLE table=test-table [id=e1d5085ac12d4bb8824dd8b2836c6a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:14.772797 27067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41b3f300eae4d268b97c510bba8d1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:14.773483 27353 tablet_bootstrap.cc:492] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15: Bootstrap starting.
I20260201 03:03:14.773976 27354 tablet_bootstrap.cc:492] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242: Bootstrap starting.
I20260201 03:03:14.774814 27353 tablet_bootstrap.cc:654] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:14.774940 27354 tablet_bootstrap.cc:654] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:14.775533 27355 tablet_bootstrap.cc:492] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f: Bootstrap starting.
I20260201 03:03:14.775923 27353 tablet_bootstrap.cc:492] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15: No bootstrap required, opened a new log
I20260201 03:03:14.775997 27353 ts_tablet_manager.cc:1403] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260201 03:03:14.776309 27355 tablet_bootstrap.cc:654] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:14.776371 27353 raft_consensus.cc:359] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } } peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } }
I20260201 03:03:14.776479 27353 raft_consensus.cc:385] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:14.776512 27353 raft_consensus.cc:740] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a481e12029469aa778691177e5ab15, State: Initialized, Role: FOLLOWER
I20260201 03:03:14.776635 27353 consensus_queue.cc:260] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } } peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } }
I20260201 03:03:14.776933 27353 ts_tablet_manager.cc:1434] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260201 03:03:14.777570 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.777194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.195:59293)
0201 03:03:14.777428 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.777485 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":113,"threads_started":1}
I20260201 03:03:14.777892 27354 tablet_bootstrap.cc:492] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242: No bootstrap required, opened a new log
I20260201 03:03:14.777984 27354 ts_tablet_manager.cc:1403] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260201 03:03:14.778221 27355 tablet_bootstrap.cc:492] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f: No bootstrap required, opened a new log
I20260201 03:03:14.778311 27355 ts_tablet_manager.cc:1403] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260201 03:03:14.778445 27354 raft_consensus.cc:359] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } } peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } }
I20260201 03:03:14.778532 27354 raft_consensus.cc:385] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:14.778561 27354 raft_consensus.cc:740] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f106a46c90471c960543c8320a0242, State: Initialized, Role: FOLLOWER
I20260201 03:03:14.778687 27354 consensus_queue.cc:260] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } } peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } }
I20260201 03:03:14.778743 27355 raft_consensus.cc:359] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } } peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } }
I20260201 03:03:14.778828 27355 raft_consensus.cc:385] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:14.778858 27355 raft_consensus.cc:740] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94bfc37fa718472db25e38cd9ad2366f, State: Initialized, Role: FOLLOWER
I20260201 03:03:14.778995 27354 ts_tablet_manager.cc:1434] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:14.778971 27355 consensus_queue.cc:260] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } } peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } }
I20260201 03:03:14.779285 27355 ts_tablet_manager.cc:1434] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:14.780171 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.779793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.194:56141)
0201 03:03:14.780020 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.780086 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
I20260201 03:03:14.781610 27135 heartbeater.cc:499] Master 127.18.203.253:38411 was elected leader, sending a full tablet report...
I20260201 03:03:14.781733 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.780933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.194:39147)
0201 03:03:14.781619 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.781669 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":620}
I20260201 03:03:14.782106 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.779044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.195:60743)
0201 03:03:14.781985 (+ 2941us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.782038 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2855}
I20260201 03:03:14.782317 27271 heartbeater.cc:499] Master 127.18.203.253:38411 was elected leader, sending a full tablet report...
I20260201 03:03:14.783061 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.779987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.193:48155)
0201 03:03:14.782883 (+ 2896us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.782938 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2750,"thread_start_us":102,"threads_started":1}
I20260201 03:03:14.783260 26998 heartbeater.cc:499] Master 127.18.203.253:38411 was elected leader, sending a full tablet report...
I20260201 03:03:14.783573 27361 raft_consensus.cc:493] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:03:14.783754 27361 raft_consensus.cc:515] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } } peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } }
I20260201 03:03:14.784794 27361 leader_election.cc:290] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94bfc37fa718472db25e38cd9ad2366f (127.18.203.194:45423), e8a481e12029469aa778691177e5ab15 (127.18.203.195:38661)
I20260201 03:03:14.785055 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.783490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.193:44913)
0201 03:03:14.784946 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.784999 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1395}
I20260201 03:03:14.787756 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.784929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:45423 (local address 127.18.203.193:46279)
0201 03:03:14.785294 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.785314 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:14.785450 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:14.785758 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:14.785761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:14.785778 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:14.786008 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:14.786014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.786771 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:14.786775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:14.787406 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:14.787414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.787502 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:14.787518 (+ 16us) client_negotiation.cc:769] Sending connection context
0201 03:03:14.787551 (+ 33us) client_negotiation.cc:240] Negotiation successful
0201 03:03:14.787594 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":108,"threads_started":1}
I20260201 03:03:14.788250 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.785245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:46279 (local address 127.18.203.194:45423)
0201 03:03:14.785363 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:14.785367 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:14.785380 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:14.785586 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:14.785590 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:14.785644 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:14.785732 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:14.786108 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.786679 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:14.787513 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.787998 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:14.788034 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:14.788079 (+ 45us) server_negotiation.cc:300] Negotiation successful
0201 03:03:14.788133 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260201 03:03:14.788596 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.785398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:59717 (local address 127.18.203.195:38661)
0201 03:03:14.785510 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:14.785514 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:14.785525 (+ 11us) server_negotiation.cc:408] Connection header received
0201 03:03:14.785564 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:14.785568 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:14.785611 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:14.785675 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:14.786342 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.786842 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:14.787745 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.788194 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:14.788224 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:14.788438 (+ 214us) server_negotiation.cc:300] Negotiation successful
0201 03:03:14.788513 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260201 03:03:14.788719 27087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41b3f300eae4d268b97c510bba8d1c3" candidate_uuid: "90f106a46c90471c960543c8320a0242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94bfc37fa718472db25e38cd9ad2366f" is_pre_election: true
I20260201 03:03:14.788851 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.784963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:38661 (local address 127.18.203.193:59717)
0201 03:03:14.785279 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.785292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:14.785384 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:14.785702 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:14.785705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:14.785724 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:14.786235 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:14.786243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.786960 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:14.786964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:14.787581 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:14.787589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.788285 (+ 696us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:14.788301 (+ 16us) client_negotiation.cc:769] Sending connection context
0201 03:03:14.788704 (+ 403us) client_negotiation.cc:240] Negotiation successful
0201 03:03:14.788767 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":83,"threads_started":1}
I20260201 03:03:14.788933 27087 raft_consensus.cc:2468] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f106a46c90471c960543c8320a0242 in term 0.
I20260201 03:03:14.789238 27223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41b3f300eae4d268b97c510bba8d1c3" candidate_uuid: "90f106a46c90471c960543c8320a0242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a481e12029469aa778691177e5ab15" is_pre_election: true
I20260201 03:03:14.789354 27223 raft_consensus.cc:2468] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f106a46c90471c960543c8320a0242 in term 0.
I20260201 03:03:14.789628 26884 leader_election.cc:304] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 90f106a46c90471c960543c8320a0242, 94bfc37fa718472db25e38cd9ad2366f, e8a481e12029469aa778691177e5ab15; no voters:
I20260201 03:03:14.789875 27365 raft_consensus.cc:2804] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:03:14.789928 27365 raft_consensus.cc:493] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:03:14.789985 27365 raft_consensus.cc:3060] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:14.790838 27365 raft_consensus.cc:515] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } } peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } }
I20260201 03:03:14.791167 27365 leader_election.cc:290] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [CANDIDATE]: Term 1 election: Requested vote from peers 94bfc37fa718472db25e38cd9ad2366f (127.18.203.194:45423), e8a481e12029469aa778691177e5ab15 (127.18.203.195:38661)
I20260201 03:03:14.791468 27087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41b3f300eae4d268b97c510bba8d1c3" candidate_uuid: "90f106a46c90471c960543c8320a0242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94bfc37fa718472db25e38cd9ad2366f"
I20260201 03:03:14.791468 27223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41b3f300eae4d268b97c510bba8d1c3" candidate_uuid: "90f106a46c90471c960543c8320a0242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a481e12029469aa778691177e5ab15"
I20260201 03:03:14.791582 27087 raft_consensus.cc:3060] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:14.791582 27223 raft_consensus.cc:3060] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:14.792626 27223 raft_consensus.cc:2468] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f106a46c90471c960543c8320a0242 in term 1.
I20260201 03:03:14.792636 27087 raft_consensus.cc:2468] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f106a46c90471c960543c8320a0242 in term 1.
I20260201 03:03:14.792888 26884 leader_election.cc:304] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90f106a46c90471c960543c8320a0242, 94bfc37fa718472db25e38cd9ad2366f; no voters:
I20260201 03:03:14.793059 27365 raft_consensus.cc:2804] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:14.793149 27365 raft_consensus.cc:697] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [term 1 LEADER]: Becoming Leader. State: Replica: 90f106a46c90471c960543c8320a0242, State: Running, Role: LEADER
I20260201 03:03:14.793524 27365 consensus_queue.cc:237] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } } peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } }
I20260201 03:03:14.793712 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.793496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.193:57193)
0201 03:03:14.793601 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.793652 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260201 03:03:14.795755 26621 catalog_manager.cc:5654] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90f106a46c90471c960543c8320a0242 (127.18.203.193). New cstate: current_term: 1 leader_uuid: "90f106a46c90471c960543c8320a0242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f106a46c90471c960543c8320a0242" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 42487 } health_report { overall_health: HEALTHY } } }
I20260201 03:03:14.812601 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.809503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.18.203.193:42487)
0201 03:03:14.809647 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:14.809652 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:14.809666 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:03:14.809790 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:14.809793 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:14.809850 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:14.809936 (+ 86us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:03:14.810307 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.810803 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:14.811462 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:14.811694 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:14.811753 (+ 59us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:03:14.812218 (+ 465us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:03:14.812335 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:14.812411 (+ 76us) server_negotiation.cc:300] Negotiation successful
0201 03:03:14.812461 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260201 03:03:14.815243 27223 raft_consensus.cc:1275] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15 [term 1 FOLLOWER]: Refusing update from remote peer 90f106a46c90471c960543c8320a0242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:14.815466 27087 raft_consensus.cc:1275] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f [term 1 FOLLOWER]: Refusing update from remote peer 90f106a46c90471c960543c8320a0242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:14.815759 27365 consensus_queue.cc:1048] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a481e12029469aa778691177e5ab15" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 38661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:14.815979 27361 consensus_queue.cc:1048] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94bfc37fa718472db25e38cd9ad2366f" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:14.818270 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.817697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.195:52773)
0201 03:03:14.818107 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.818183 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
I20260201 03:03:14.819399 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.818009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.194:41545)
0201 03:03:14.819264 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.819325 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1178}
I20260201 03:03:14.821753 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.821380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.195:51837)
0201 03:03:14.821631 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.821684 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260201 03:03:14.821965 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.821444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.193:57105)
0201 03:03:14.821842 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.821893 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329}
I20260201 03:03:14.822458 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.822229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:40003 (local address 127.18.203.194:34013)
0201 03:03:14.822345 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.822394 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260201 03:03:14.824980 27370 mvcc.cc:204] Tried to move back new op lower bound from 7249571818758873088 to 7249571818676678656. Current Snapshot: MvccSnapshot[applied={T|T < 7249571818758873088 or (T in {7249571818758873088})}]
I20260201 03:03:14.828855 26620 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:14.829013 26620 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:14.832791 26620 catalog_manager.cc:5941] T 00000000000000000000000000000000 P d3a7a00d1b784a3fa9cd98e967c8eed4: Sending DeleteTablet for 3 replicas of tablet f41b3f300eae4d268b97c510bba8d1c3
I20260201 03:03:14.833451 27203 tablet_service.cc:1552] Processing DeleteTablet for tablet f41b3f300eae4d268b97c510bba8d1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:14 UTC) from {username='oryx', principal='oryx/127.18.203.253@KRBTEST.COM'} at 127.0.0.1:51300
I20260201 03:03:14.833644 27067 tablet_service.cc:1552] Processing DeleteTablet for tablet f41b3f300eae4d268b97c510bba8d1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:14 UTC) from {username='oryx', principal='oryx/127.18.203.253@KRBTEST.COM'} at 127.0.0.1:38250
I20260201 03:03:14.833796 26930 tablet_service.cc:1552] Processing DeleteTablet for tablet f41b3f300eae4d268b97c510bba8d1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:14 UTC) from {username='oryx', principal='oryx/127.18.203.253@KRBTEST.COM'} at 127.0.0.1:40284
I20260201 03:03:14.834133 27381 tablet_replica.cc:333] T f41b3f300eae4d268b97c510bba8d1c3 P 90f106a46c90471c960543c8320a0242: stopping tablet replica
I20260201 03:03:14.834419 27380 tablet_replica.cc:333] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f: stopping tablet replica
I20260201 03:03:14.834434 27379 tablet_replica.cc:333] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15: stopping tablet replica
I20260201 03:03:14.834542 27380 raft_consensus.cc:2243] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:14.834678 27380 raft_consensus.cc:2272] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:14.834669 27379 raft_consensus.cc:2243] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:14.834857 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 26867
I20260201 03:03:14.834897 27379 raft_consensus.cc:2272] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:14.835638 27380 ts_tablet_manager.cc:1916] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:14.835860 27379 ts_tablet_manager.cc:1916] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:14.838059 27379 ts_tablet_manager.cc:1929] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:14.838099 27380 ts_tablet_manager.cc:1929] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:14.838130 27379 log.cc:1199] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/f41b3f300eae4d268b97c510bba8d1c3
I20260201 03:03:14.838164 27380 log.cc:1199] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/f41b3f300eae4d268b97c510bba8d1c3
I20260201 03:03:14.838392 27379 ts_tablet_manager.cc:1950] T f41b3f300eae4d268b97c510bba8d1c3 P e8a481e12029469aa778691177e5ab15: Deleting consensus metadata
I20260201 03:03:14.838454 27380 ts_tablet_manager.cc:1950] T f41b3f300eae4d268b97c510bba8d1c3 P 94bfc37fa718472db25e38cd9ad2366f: Deleting consensus metadata
I20260201 03:03:14.839088 26607 catalog_manager.cc:4985] TS e8a481e12029469aa778691177e5ab15 (127.18.203.195:38661): tablet f41b3f300eae4d268b97c510bba8d1c3 (table test-table [id=e1d5085ac12d4bb8824dd8b2836c6a44]) successfully deleted
I20260201 03:03:14.839092 26606 catalog_manager.cc:4985] TS 94bfc37fa718472db25e38cd9ad2366f (127.18.203.194:45423): tablet f41b3f300eae4d268b97c510bba8d1c3 (table test-table [id=e1d5085ac12d4bb8824dd8b2836c6a44]) successfully deleted
I20260201 03:03:14.843660 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 27003
I20260201 03:03:14.843945 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:14.843691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:42487 (local address 127.0.0.1:40302)
0201 03:03:14.843812 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:14.843861 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.193:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260201 03:03:14.844461 26607 catalog_manager.cc:4712] TS 90f106a46c90471c960543c8320a0242 (127.18.203.193:42487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f41b3f300eae4d268b97c510bba8d1c3: Network error: Client connection negotiation failed: client connection to 127.18.203.193:42487: connect: Connection refused (error 111)
I20260201 03:03:14.852886 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 27140
I20260201 03:03:14.858546 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 26480
I20260201 03:03:14.864812 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 26593
2026-02-01T03:03:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7073 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:03:14 dist-test-slave-gf59 krb5kdc[27384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:03:14 dist-test-slave-gf59 krb5kdc[27384](info): set up 2 sockets
Feb 01 03:03:14 dist-test-slave-gf59 krb5kdc[27384](info): commencing operation
krb5kdc: starting...
W20260201 03:03:16.688333 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.800s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:03:16 dist-test-slave-gf59 krb5kdc[27384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:03:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:03:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
I20260201 03:03:16.835284 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:44641
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:44641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:16.945559 27400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:16.945822 27400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:16.945892 27400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:16.949224 27400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:16.949301 27400 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260201 03:03:16.949319 27400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:16.949357 27400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:16.949391 27400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:16.949421 27400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:03:16.953840 27400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:44641
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:44641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:16.955027 27400 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:16.955986 27400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:16.962349 27408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:16.962321 27405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:16.962349 27406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:16.962491 27400 server_base.cc:1060] running on GCE node
I20260201 03:03:16.963181 27400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:16.964185 27400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:16.965335 27400 hybrid_clock.cc:648] HybridClock initialized: now 1769914996965316 us; error 31 us; skew 500 ppm
Feb 01 03:03:16 dist-test-slave-gf59 krb5kdc[27384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914996, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:16.968011 27400 init.cc:377] Logged in from keytab as oryx/127.18.203.254@KRBTEST.COM (short username oryx)
I20260201 03:03:16.969146 27400 webserver.cc:492] Webserver started at http://127.18.203.254:45535/ using document root <none> and password file <none>
I20260201 03:03:16.969691 27400 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:16.969743 27400 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:16.969944 27400 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:16.971801 27400 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "b606226d8d014ecbbb0f1f1d27f7b785"
format_stamp: "Formatted at 2026-02-01 03:03:16 on dist-test-slave-gf59"
server_key: "abe885da5d164399992f8ea02118005c"
server_key_iv: "b2a3d058d25d45c11f92b14641657882"
server_key_version: "encryptionkey@0"
I20260201 03:03:16.972298 27400 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "b606226d8d014ecbbb0f1f1d27f7b785"
format_stamp: "Formatted at 2026-02-01 03:03:16 on dist-test-slave-gf59"
server_key: "abe885da5d164399992f8ea02118005c"
server_key_iv: "b2a3d058d25d45c11f92b14641657882"
server_key_version: "encryptionkey@0"
I20260201 03:03:16.975788 27400 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260201 03:03:16.978067 27415 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:16.979137 27400 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:16.979251 27400 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "b606226d8d014ecbbb0f1f1d27f7b785"
format_stamp: "Formatted at 2026-02-01 03:03:16 on dist-test-slave-gf59"
server_key: "abe885da5d164399992f8ea02118005c"
server_key_iv: "b2a3d058d25d45c11f92b14641657882"
server_key_version: "encryptionkey@0"
I20260201 03:03:16.979344 27400 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:16.995429 27400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:16.998443 27400 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:16.998631 27400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:17.006804 27400 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:44641
I20260201 03:03:17.007025 27477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:44641 every 8 connection(s)
I20260201 03:03:17.007942 27400 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:03:17.010763 27478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:17.011873 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 27400
I20260201 03:03:17.011986 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:03:17.012214 19247 external_mini_cluster.cc:1468] Setting key 81c2aff0773c69b3b305a48a0b322a76
I20260201 03:03:17.016858 27478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785: Bootstrap starting.
I20260201 03:03:17.019196 27478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785: Neither blocks nor log segments found. Creating new log.
Feb 01 03:03:17 dist-test-slave-gf59 krb5kdc[27384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:17.020076 27478 log.cc:826] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:17.022127 27478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785: No bootstrap required, opened a new log
I20260201 03:03:17.025048 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.013494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.18.203.254:44641)
0201 03:03:17.013925 (+ 431us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:17.013936 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:17.013978 (+ 42us) server_negotiation.cc:408] Connection header received
0201 03:03:17.014624 (+ 646us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:17.014651 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:17.015026 (+ 375us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:17.015401 (+ 375us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:17.016209 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.017317 (+ 1108us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.017972 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.018254 (+ 282us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.020458 (+ 2204us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:17.020477 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:17.020490 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:17.020522 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:17.022396 (+ 1874us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.022778 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.022783 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.022789 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.022899 (+ 110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.023084 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.023087 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.023089 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.023531 (+ 442us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:17.023779 (+ 248us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:17.023968 (+ 189us) server_negotiation.cc:300] Negotiation successful
0201 03:03:17.024230 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":123,"threads_started":1}
I20260201 03:03:17.025871 27478 raft_consensus.cc:359] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b606226d8d014ecbbb0f1f1d27f7b785" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 44641 } }
I20260201 03:03:17.026098 27478 raft_consensus.cc:385] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:17.026160 27478 raft_consensus.cc:740] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b606226d8d014ecbbb0f1f1d27f7b785, State: Initialized, Role: FOLLOWER
I20260201 03:03:17.026593 27478 consensus_queue.cc:260] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b606226d8d014ecbbb0f1f1d27f7b785" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 44641 } }
I20260201 03:03:17.026726 27478 raft_consensus.cc:399] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:03:17.026785 27478 raft_consensus.cc:493] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:03:17.026875 27478 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:17.027797 27478 raft_consensus.cc:515] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b606226d8d014ecbbb0f1f1d27f7b785" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 44641 } }
I20260201 03:03:17.028146 27478 leader_election.cc:304] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b606226d8d014ecbbb0f1f1d27f7b785; no voters:
I20260201 03:03:17.028429 27478 leader_election.cc:290] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:03:17.028524 27483 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:17.028731 27483 raft_consensus.cc:697] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [term 1 LEADER]: Becoming Leader. State: Replica: b606226d8d014ecbbb0f1f1d27f7b785, State: Running, Role: LEADER
I20260201 03:03:17.029003 27483 consensus_queue.cc:237] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b606226d8d014ecbbb0f1f1d27f7b785" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 44641 } }
I20260201 03:03:17.029502 27478 sys_catalog.cc:565] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:17.030580 27484 sys_catalog.cc:455] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b606226d8d014ecbbb0f1f1d27f7b785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b606226d8d014ecbbb0f1f1d27f7b785" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 44641 } } }
I20260201 03:03:17.030699 27484 sys_catalog.cc:458] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:17.030930 27485 sys_catalog.cc:455] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b606226d8d014ecbbb0f1f1d27f7b785. Latest consensus state: current_term: 1 leader_uuid: "b606226d8d014ecbbb0f1f1d27f7b785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b606226d8d014ecbbb0f1f1d27f7b785" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 44641 } } }
I20260201 03:03:17.031013 27485 sys_catalog.cc:458] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:17.031147 27492 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:03:17.034191 27492 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:03:17.039769 27492 catalog_manager.cc:1357] Generated new cluster ID: 4411e01c14984e0abbbb17aef05ae76f
I20260201 03:03:17.039853 27492 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:03:17.048857 27492 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:03:17.049875 27492 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:03:17.058723 27492 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785: Generated new TSK 0
I20260201 03:03:17.059540 27492 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
I20260201 03:03:17.116714 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:44641
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:17.229215 27506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:17.229472 27506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:17.229532 27506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:17.232926 27506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:17.232997 27506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:17.233058 27506 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:03:17.233116 27506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:03:17.237468 27506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.203.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:17.238602 27506 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:17.239490 27506 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:17.246232 27514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:17.246274 27512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:17.246234 27511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:17.246621 27506 server_base.cc:1060] running on GCE node
I20260201 03:03:17.246986 27506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:17.247548 27506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:17.248697 27506 hybrid_clock.cc:648] HybridClock initialized: now 1769914997248684 us; error 28 us; skew 500 ppm
Feb 01 03:03:17 dist-test-slave-gf59 krb5kdc[27384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:17.251259 27506 init.cc:377] Logged in from keytab as oryx/127.18.203.193@KRBTEST.COM (short username oryx)
I20260201 03:03:17.252444 27506 webserver.cc:492] Webserver started at http://127.18.203.193:41647/ using document root <none> and password file <none>
I20260201 03:03:17.252992 27506 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:17.253046 27506 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:17.253233 27506 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:17.254967 27506 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "8c779a097be24011a4b5fb6ddf86a532"
format_stamp: "Formatted at 2026-02-01 03:03:17 on dist-test-slave-gf59"
server_key: "a894166e0b3c7bf320d15ad66c2dd950"
server_key_iv: "f6e781a3868124ba9c917333eafc4365"
server_key_version: "encryptionkey@0"
I20260201 03:03:17.255436 27506 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "8c779a097be24011a4b5fb6ddf86a532"
format_stamp: "Formatted at 2026-02-01 03:03:17 on dist-test-slave-gf59"
server_key: "a894166e0b3c7bf320d15ad66c2dd950"
server_key_iv: "f6e781a3868124ba9c917333eafc4365"
server_key_version: "encryptionkey@0"
I20260201 03:03:17.258891 27506 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260201 03:03:17.261080 27521 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:17.262125 27506 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:17.262260 27506 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "8c779a097be24011a4b5fb6ddf86a532"
format_stamp: "Formatted at 2026-02-01 03:03:17 on dist-test-slave-gf59"
server_key: "a894166e0b3c7bf320d15ad66c2dd950"
server_key_iv: "f6e781a3868124ba9c917333eafc4365"
server_key_version: "encryptionkey@0"
I20260201 03:03:17.262365 27506 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:17.281589 27506 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:17.284924 27506 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:17.285152 27506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:17.286767 27506 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:17.286829 27506 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:17.286888 27506 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:17.286929 27506 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:03:17 dist-test-slave-gf59 krb5kdc[27384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:17.298671 27506 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:43881
I20260201 03:03:17.298699 27636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:43881 every 8 connection(s)
I20260201 03:03:17.298969 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.287463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:37919 (local address 127.18.203.254:44641)
0201 03:03:17.287584 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:17.287588 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:17.288516 (+ 928us) server_negotiation.cc:408] Connection header received
0201 03:03:17.289395 (+ 879us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:17.289399 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:17.289462 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:17.289538 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:17.291248 (+ 1710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.292123 (+ 875us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.292912 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.293103 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.295528 (+ 2425us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:17.295550 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:17.295552 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:17.295583 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:17.297065 (+ 1482us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.297538 (+ 473us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.297541 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.297542 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.297593 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.297843 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.297847 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.297849 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.298014 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:17.298110 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:17.298657 (+ 547us) server_negotiation.cc:300] Negotiation successful
0201 03:03:17.298806 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260201 03:03:17.299569 27530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.287875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:44641 (local address 127.18.203.193:37919)
0201 03:03:17.288382 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:17.288419 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:17.289192 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:17.289714 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:17.289723 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:17.290306 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:17.291062 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.291076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.292262 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.292266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:17.292798 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.292806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.293031 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.293699 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:17.293757 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:17.295353 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:17.297180 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:17.297186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:17.297195 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:17.297441 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:17.297672 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:17.297675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:17.297677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:17.297761 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:17.298114 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:17.298121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:17.298374 (+ 253us) client_negotiation.cc:769] Sending connection context
0201 03:03:17.298599 (+ 225us) client_negotiation.cc:240] Negotiation successful
0201 03:03:17.298870 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":156,"threads_started":1}
I20260201 03:03:17.299751 27506 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:03:17.301614 27637 heartbeater.cc:344] Connected to a master server at 127.18.203.254:44641
I20260201 03:03:17.301765 27637 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:17.302331 27637 heartbeater.cc:507] Master 127.18.203.254:44641 requested a full tablet report, sending...
I20260201 03:03:17.303805 27432 ts_manager.cc:194] Registered new tserver with Master: 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193:43881)
I20260201 03:03:17.304116 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 27506
I20260201 03:03:17.304241 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:03:17.304514 19247 external_mini_cluster.cc:1468] Setting key 82be3c44211651d90afb70fc4607f37a
I20260201 03:03:17.305213 27432 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:37919
I20260201 03:03:17.310526 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.303104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:37203 (local address 127.18.203.254:44641)
0201 03:03:17.303225 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:17.303229 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:17.304115 (+ 886us) server_negotiation.cc:408] Connection header received
0201 03:03:17.304409 (+ 294us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:17.304413 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:17.304457 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:17.304501 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:17.305261 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.305797 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.306490 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.306663 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.307688 (+ 1025us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:17.307703 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:17.307705 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:17.307728 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:17.309091 (+ 1363us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.309484 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.309488 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.309490 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.309537 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.309756 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.309759 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.309762 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.309908 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:17.310010 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:17.310264 (+ 254us) server_negotiation.cc:300] Negotiation successful
0201 03:03:17.310377 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260201 03:03:17.310549 27643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.303036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:44641 (local address 127.18.203.193:37203)
0201 03:03:17.304035 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:17.304051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:17.304263 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:17.304669 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:17.304671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:17.304833 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:17.305119 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.305129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.305916 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.305919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:17.306384 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.306392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.306496 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.307205 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:17.307234 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:17.307530 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:17.309205 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:17.309209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:17.309212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:17.309395 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:17.309622 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:17.309625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:17.309628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:17.309675 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:17.310009 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:17.310012 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:17.310098 (+ 86us) client_negotiation.cc:769] Sending connection context
0201 03:03:17.310193 (+ 95us) client_negotiation.cc:240] Negotiation successful
0201 03:03:17.310339 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
I20260201 03:03:17.352790 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:44641
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:17.461949 27648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:17.462205 27648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:17.462275 27648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:17.465658 27648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:17.465739 27648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:17.465795 27648 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:03:17.465857 27648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:03:17.470140 27648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.203.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:17.471248 27648 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:17.472142 27648 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:17.478811 27654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:17.478842 27653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:17.478933 27648 server_base.cc:1060] running on GCE node
W20260201 03:03:17.478807 27656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:17.479444 27648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:17.480017 27648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:17.481215 27648 hybrid_clock.cc:648] HybridClock initialized: now 1769914997481198 us; error 28 us; skew 500 ppm
Feb 01 03:03:17 dist-test-slave-gf59 krb5kdc[27384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:17.484120 27648 init.cc:377] Logged in from keytab as oryx/127.18.203.194@KRBTEST.COM (short username oryx)
I20260201 03:03:17.485289 27648 webserver.cc:492] Webserver started at http://127.18.203.194:41021/ using document root <none> and password file <none>
I20260201 03:03:17.485842 27648 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:17.485909 27648 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:17.486116 27648 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:17.487946 27648 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "02b549b583fb400a89689579b2caa287"
format_stamp: "Formatted at 2026-02-01 03:03:17 on dist-test-slave-gf59"
server_key: "f0ef158bffc3fba458a656ee269f17b6"
server_key_iv: "263c490f34b2f34d4288c0e9a7af8806"
server_key_version: "encryptionkey@0"
I20260201 03:03:17.488471 27648 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "02b549b583fb400a89689579b2caa287"
format_stamp: "Formatted at 2026-02-01 03:03:17 on dist-test-slave-gf59"
server_key: "f0ef158bffc3fba458a656ee269f17b6"
server_key_iv: "263c490f34b2f34d4288c0e9a7af8806"
server_key_version: "encryptionkey@0"
I20260201 03:03:17.492007 27648 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260201 03:03:17.494277 27663 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:17.495363 27648 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:17.495527 27648 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "02b549b583fb400a89689579b2caa287"
format_stamp: "Formatted at 2026-02-01 03:03:17 on dist-test-slave-gf59"
server_key: "f0ef158bffc3fba458a656ee269f17b6"
server_key_iv: "263c490f34b2f34d4288c0e9a7af8806"
server_key_version: "encryptionkey@0"
I20260201 03:03:17.495667 27648 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:17.509562 27648 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:17.515261 27648 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:17.515475 27648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:17.517155 27648 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:17.517231 27648 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:17.517289 27648 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:17.517323 27648 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:03:17 dist-test-slave-gf59 krb5kdc[27384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:17.528704 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.517735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:51323 (local address 127.18.203.254:44641)
0201 03:03:17.517864 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:17.517868 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:17.518747 (+ 879us) server_negotiation.cc:408] Connection header received
0201 03:03:17.519706 (+ 959us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:17.519711 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:17.519780 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:17.519887 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:17.521345 (+ 1458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.521861 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.522742 (+ 881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.522994 (+ 252us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.525461 (+ 2467us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:17.525485 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:17.525488 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:17.525524 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:17.526880 (+ 1356us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.527329 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.527333 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.527335 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.527387 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.527655 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.527659 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.527661 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.527826 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:17.527914 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:17.528411 (+ 497us) server_negotiation.cc:300] Negotiation successful
0201 03:03:17.528545 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260201 03:03:17.528910 27648 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:41813
I20260201 03:03:17.529359 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.518121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:44641 (local address 127.18.203.194:51323)
0201 03:03:17.518600 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:17.518639 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:17.519459 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:17.520046 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:17.520055 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:17.520533 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:17.521194 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.521212 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.521971 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.521975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:17.522583 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.522594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.522825 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.523529 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:17.523566 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:17.525300 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:17.526981 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:17.526987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:17.526999 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:17.527236 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:17.527469 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:17.527472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:17.527474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:17.527557 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:17.527922 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:17.527929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:17.528172 (+ 243us) client_negotiation.cc:769] Sending connection context
0201 03:03:17.528365 (+ 193us) client_negotiation.cc:240] Negotiation successful
0201 03:03:17.528632 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":145,"threads_started":1}
I20260201 03:03:17.530068 27648 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:03:17.530911 27778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:41813 every 8 connection(s)
I20260201 03:03:17.531445 27779 heartbeater.cc:344] Connected to a master server at 127.18.203.254:44641
I20260201 03:03:17.531637 27779 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:17.532194 27779 heartbeater.cc:507] Master 127.18.203.254:44641 requested a full tablet report, sending...
I20260201 03:03:17.533172 27431 ts_manager.cc:194] Registered new tserver with Master: 02b549b583fb400a89689579b2caa287 (127.18.203.194:41813)
I20260201 03:03:17.533946 27431 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:51323
I20260201 03:03:17.538952 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.532859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:34071 (local address 127.18.203.254:44641)
0201 03:03:17.532980 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:17.532983 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:17.533159 (+ 176us) server_negotiation.cc:408] Connection header received
0201 03:03:17.533342 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:17.533346 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:17.533388 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:17.533451 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:17.534113 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.534700 (+ 587us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.535425 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.535554 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.536423 (+ 869us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:17.536437 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:17.536440 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:17.536461 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:17.537617 (+ 1156us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.538016 (+ 399us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.538019 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.538020 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.538055 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.538277 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.538279 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.538280 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.538439 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:17.538511 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:17.538753 (+ 242us) server_negotiation.cc:300] Negotiation successful
0201 03:03:17.538843 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260201 03:03:17.538968 27785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.532782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:44641 (local address 127.18.203.194:34071)
0201 03:03:17.533073 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:17.533089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:17.533222 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:17.533490 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:17.533494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:17.533697 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:17.533982 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.533993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.534813 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.534818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:17.535318 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.535328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.535447 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.535956 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:17.535971 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:17.536319 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:17.537732 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:17.537735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:17.537737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:17.537917 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:17.538149 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:17.538152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:17.538153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:17.538205 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:17.538536 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:17.538539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:17.538622 (+ 83us) client_negotiation.cc:769] Sending connection context
0201 03:03:17.538719 (+ 97us) client_negotiation.cc:240] Negotiation successful
0201 03:03:17.538825 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
I20260201 03:03:17.539320 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 27648
I20260201 03:03:17.539438 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:03:17.539734 19247 external_mini_cluster.cc:1468] Setting key dac53fa1d5e9d18e728c7cc40cb53d9c
WARNING: no policy specified for oryx/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab.
I20260201 03:03:17.588639 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:44641
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:17.697535 27790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:17.697893 27790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:17.698019 27790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:17.701495 27790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:17.701574 27790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:17.701651 27790 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260201 03:03:17.701722 27790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:03:17.706149 27790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.203.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:17.707298 27790 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:17.708240 27790 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:17.715397 27796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:17.715400 27795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:17.715399 27798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:17.715716 27790 server_base.cc:1060] running on GCE node
I20260201 03:03:17.716192 27790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:17.716764 27790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:17.717916 27790 hybrid_clock.cc:648] HybridClock initialized: now 1769914997717904 us; error 35 us; skew 500 ppm
Feb 01 03:03:17 dist-test-slave-gf59 krb5kdc[27384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:17.720621 27790 init.cc:377] Logged in from keytab as oryx/127.18.203.195@KRBTEST.COM (short username oryx)
I20260201 03:03:17.721773 27790 webserver.cc:492] Webserver started at http://127.18.203.195:45671/ using document root <none> and password file <none>
I20260201 03:03:17.722361 27790 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:17.722417 27790 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:17.722601 27790 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:17.724457 27790 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "273bae1caac44d809d0ea8a3417710b1"
format_stamp: "Formatted at 2026-02-01 03:03:17 on dist-test-slave-gf59"
server_key: "7a2271ddd77c90bd4f0c369523b70c99"
server_key_iv: "ab693551ca2a4a340daf61589cfb21a9"
server_key_version: "encryptionkey@0"
I20260201 03:03:17.724920 27790 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "273bae1caac44d809d0ea8a3417710b1"
format_stamp: "Formatted at 2026-02-01 03:03:17 on dist-test-slave-gf59"
server_key: "7a2271ddd77c90bd4f0c369523b70c99"
server_key_iv: "ab693551ca2a4a340daf61589cfb21a9"
server_key_version: "encryptionkey@0"
I20260201 03:03:17.728279 27790 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:03:17.730445 27805 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:17.731496 27790 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260201 03:03:17.731602 27790 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "273bae1caac44d809d0ea8a3417710b1"
format_stamp: "Formatted at 2026-02-01 03:03:17 on dist-test-slave-gf59"
server_key: "7a2271ddd77c90bd4f0c369523b70c99"
server_key_iv: "ab693551ca2a4a340daf61589cfb21a9"
server_key_version: "encryptionkey@0"
I20260201 03:03:17.731729 27790 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:17.746461 27790 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:17.749104 27790 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:17.749286 27790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:17.750804 27790 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:17.750905 27790 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:17.750990 27790 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:17.751039 27790 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 01 03:03:17 dist-test-slave-gf59 krb5kdc[27384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:17.762117 27790 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:33317
I20260201 03:03:17.762146 27920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:33317 every 8 connection(s)
I20260201 03:03:17.762956 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.751359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:41601 (local address 127.18.203.254:44641)
0201 03:03:17.751493 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:17.751497 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:17.752388 (+ 891us) server_negotiation.cc:408] Connection header received
0201 03:03:17.753239 (+ 851us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:17.753243 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:17.753317 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:17.753395 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:17.754789 (+ 1394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.755416 (+ 627us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.756345 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.756548 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.759090 (+ 2542us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:17.759112 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:17.759114 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:17.759143 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:17.760683 (+ 1540us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.761250 (+ 567us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.761252 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.761254 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.761306 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.761737 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.761741 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.761743 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.761968 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:17.762080 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:17.762648 (+ 568us) server_negotiation.cc:300] Negotiation successful
0201 03:03:17.762779 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260201 03:03:17.763326 27790 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:03:17.763528 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.751765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:44641 (local address 127.18.203.195:41601)
0201 03:03:17.752247 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:17.752283 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:17.753028 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:17.753570 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:17.753579 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:17.754074 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:17.754634 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.754648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.755548 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.755553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:17.756206 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.756219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.756479 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.757120 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:17.757161 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:17.758916 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:17.760810 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:17.760816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:17.760828 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:17.761153 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:17.761439 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:17.761443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:17.761445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:17.761597 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:17.762112 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:17.762120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:17.762399 (+ 279us) client_negotiation.cc:769] Sending connection context
0201 03:03:17.762578 (+ 179us) client_negotiation.cc:240] Negotiation successful
0201 03:03:17.762797 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":134,"threads_started":1}
I20260201 03:03:17.764858 27921 heartbeater.cc:344] Connected to a master server at 127.18.203.254:44641
I20260201 03:03:17.765043 27921 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:17.765233 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 27790
I20260201 03:03:17.765344 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:03:17.765620 19247 external_mini_cluster.cc:1468] Setting key 50085bf7fd56ba9765261cbf099d26b3
I20260201 03:03:17.765640 27921 heartbeater.cc:507] Master 127.18.203.254:44641 requested a full tablet report, sending...
I20260201 03:03:17.766638 27431 ts_manager.cc:194] Registered new tserver with Master: 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317)
I20260201 03:03:17.767279 27431 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:41601
I20260201 03:03:17.767911 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260201 03:03:17.774412 27927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.766883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:44641 (local address 127.18.203.195:50159)
0201 03:03:17.767118 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:17.767133 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:17.767216 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:17.769147 (+ 1931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:17.769150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:17.769309 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:17.769513 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.769519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.770298 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.770302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:17.770674 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.770681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.770773 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.771259 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:17.771274 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:17.771652 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:17.773162 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:17.773166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:17.773168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:17.773381 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:17.773603 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:17.773606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:17.773607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:17.773655 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:17.773985 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:17.773987 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:17.774090 (+ 103us) client_negotiation.cc:769] Sending connection context
0201 03:03:17.774167 (+ 77us) client_negotiation.cc:240] Negotiation successful
0201 03:03:17.774270 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260201 03:03:17.774494 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.767526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:50159 (local address 127.18.203.254:44641)
0201 03:03:17.767658 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:17.767662 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:17.767675 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:17.767709 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:17.767712 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:17.767751 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:17.767819 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:17.769628 (+ 1809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.770193 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.770773 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.770907 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.771761 (+ 854us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:17.771775 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:17.771778 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:17.771800 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:17.773058 (+ 1258us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.773470 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.773473 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.773475 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.773513 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.773730 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.773733 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.773735 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.773906 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:17.774001 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:17.774231 (+ 230us) server_negotiation.cc:300] Negotiation successful
0201 03:03:17.774351 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Feb 01 03:03:17 dist-test-slave-gf59 krb5kdc[27384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914997, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 01 03:03:17 dist-test-slave-gf59 krb5kdc[27384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:17 dist-test-slave-gf59 krb5kdc[27384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
W20260201 03:03:17.791565 27481 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:17.786375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.18.203.254:44641)
0201 03:03:17.786510 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:17.786514 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:17.786526 (+ 12us) server_negotiation.cc:408] Connection header received
0201 03:03:17.786565 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:17.786568 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:17.786613 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:17.786682 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:17.787444 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.787959 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.788572 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.788693 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.791413 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53406: BlockingRecv error: recv got EOF from 127.0.0.1:53406 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Feb 01 03:03:17 dist-test-slave-gf59 krb5kdc[27384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914997, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:17.799508 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.792924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.18.203.254:44641)
0201 03:03:17.793058 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:17.793063 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:17.793131 (+ 68us) server_negotiation.cc:408] Connection header received
0201 03:03:17.793255 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:17.793258 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:17.793298 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:17.793363 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:17.794006 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.794478 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.795042 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.795222 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.797082 (+ 1860us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:17.797096 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:17.797098 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:17.797124 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:17.798219 (+ 1095us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.798566 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.798570 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.798571 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.798613 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:17.798799 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:17.798802 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:17.798804 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:17.798942 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:17.799067 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:17.799249 (+ 182us) server_negotiation.cc:300] Negotiation successful
0201 03:03:17.799355 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260201 03:03:17.802016 27432 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:03:17.804459 27432 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:03:17.817181 27943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.812494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:33317 (local address 127.0.0.1:41858)
0201 03:03:17.812815 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:17.812860 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:17.813016 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:17.813811 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:17.813817 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:17.813843 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:17.814189 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.814199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.815773 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.815778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:17.816656 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.816665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.816757 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.816818 (+ 61us) client_negotiation.cc:769] Sending connection context
0201 03:03:17.816911 (+ 93us) client_negotiation.cc:240] Negotiation successful
0201 03:03:17.817014 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"spinlock_wait_cycles":768,"thread_start_us":76,"threads_started":1}
I20260201 03:03:17.817248 27945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.812703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:41813 (local address 127.0.0.1:39382)
0201 03:03:17.813798 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:17.813812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:17.813917 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:17.814398 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:17.814401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:17.814420 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:17.814673 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.814681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.815966 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.815969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:17.816847 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.816856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.816960 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.816990 (+ 30us) client_negotiation.cc:769] Sending connection context
0201 03:03:17.817038 (+ 48us) client_negotiation.cc:240] Negotiation successful
0201 03:03:17.817098 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"spinlock_wait_cycles":896,"thread_start_us":95,"threads_started":1}
I20260201 03:03:17.817937 27944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.812626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.18.203.195:33317)
0201 03:03:17.813316 (+ 690us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:17.813324 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:17.813346 (+ 22us) server_negotiation.cc:408] Connection header received
0201 03:03:17.813425 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:17.813430 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:17.813619 (+ 189us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:17.814333 (+ 714us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:17.814494 (+ 161us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.815596 (+ 1102us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.816941 (+ 1345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.817611 (+ 670us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.817710 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:17.817764 (+ 54us) server_negotiation.cc:300] Negotiation successful
0201 03:03:17.817831 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":140,"threads_started":1}
I20260201 03:03:17.817251 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.813441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:43881 (local address 127.0.0.1:40060)
0201 03:03:17.814223 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:17.814235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:17.814336 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:17.814745 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:17.814748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:17.814764 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:17.814962 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.814969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.816100 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.816103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:17.816929 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:17.816936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.817019 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.817031 (+ 12us) client_negotiation.cc:769] Sending connection context
0201 03:03:17.817073 (+ 42us) client_negotiation.cc:240] Negotiation successful
0201 03:03:17.817115 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":71,"threads_started":1}
I20260201 03:03:17.818421 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.813272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.18.203.194:41813)
0201 03:03:17.813665 (+ 393us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:17.813671 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:17.814004 (+ 333us) server_negotiation.cc:408] Connection header received
0201 03:03:17.814103 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:17.814109 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:17.814278 (+ 169us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:17.814416 (+ 138us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:17.814802 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.815862 (+ 1060us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.817565 (+ 1703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.818055 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.818147 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:17.818203 (+ 56us) server_negotiation.cc:300] Negotiation successful
0201 03:03:17.818286 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":98,"threads_started":1}
I20260201 03:03:17.818778 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:17.813843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.18.203.193:43881)
0201 03:03:17.814178 (+ 335us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:17.814183 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:17.814319 (+ 136us) server_negotiation.cc:408] Connection header received
0201 03:03:17.814495 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:17.814500 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:17.814631 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:17.814739 (+ 108us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:17.815072 (+ 333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.815990 (+ 918us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:17.817307 (+ 1317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:17.817805 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:17.817913 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:17.817978 (+ 65us) server_negotiation.cc:300] Negotiation successful
0201 03:03:17.818648 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":75,"threads_started":1}
I20260201 03:03:17.821002 27713 tablet_service.cc:1505] Processing CreateTablet for tablet f19b65a58cde46e3a92f6713bf6b3a79 (DEFAULT_TABLE table=test-table [id=8a775af0a4204eab83a2464c662af54a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:17.821007 27855 tablet_service.cc:1505] Processing CreateTablet for tablet f19b65a58cde46e3a92f6713bf6b3a79 (DEFAULT_TABLE table=test-table [id=8a775af0a4204eab83a2464c662af54a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:17.821017 27571 tablet_service.cc:1505] Processing CreateTablet for tablet f19b65a58cde46e3a92f6713bf6b3a79 (DEFAULT_TABLE table=test-table [id=8a775af0a4204eab83a2464c662af54a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:17.821873 27713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f19b65a58cde46e3a92f6713bf6b3a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:17.821878 27571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f19b65a58cde46e3a92f6713bf6b3a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:17.821878 27855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f19b65a58cde46e3a92f6713bf6b3a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:17.827190 27949 tablet_bootstrap.cc:492] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287: Bootstrap starting.
I20260201 03:03:17.828668 27950 tablet_bootstrap.cc:492] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532: Bootstrap starting.
I20260201 03:03:17.830173 27949 tablet_bootstrap.cc:654] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:17.830617 27951 tablet_bootstrap.cc:492] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1: Bootstrap starting.
I20260201 03:03:17.830848 27950 tablet_bootstrap.cc:654] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:17.831547 27949 log.cc:826] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:17.831547 27950 log.cc:826] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:17.832537 27951 tablet_bootstrap.cc:654] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:17.833490 27951 log.cc:826] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:17.833525 27950 tablet_bootstrap.cc:492] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532: No bootstrap required, opened a new log
I20260201 03:03:17.833526 27949 tablet_bootstrap.cc:492] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287: No bootstrap required, opened a new log
I20260201 03:03:17.833704 27949 ts_tablet_manager.cc:1403] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260201 03:03:17.833704 27950 ts_tablet_manager.cc:1403] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260201 03:03:17.835407 27951 tablet_bootstrap.cc:492] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1: No bootstrap required, opened a new log
I20260201 03:03:17.835584 27951 ts_tablet_manager.cc:1403] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260201 03:03:17.836876 27949 raft_consensus.cc:359] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } }
I20260201 03:03:17.836876 27950 raft_consensus.cc:359] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } }
I20260201 03:03:17.837105 27950 raft_consensus.cc:385] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:17.837105 27949 raft_consensus.cc:385] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:17.837172 27950 raft_consensus.cc:740] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c779a097be24011a4b5fb6ddf86a532, State: Initialized, Role: FOLLOWER
I20260201 03:03:17.837172 27949 raft_consensus.cc:740] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b549b583fb400a89689579b2caa287, State: Initialized, Role: FOLLOWER
I20260201 03:03:17.837637 27949 consensus_queue.cc:260] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } }
I20260201 03:03:17.837637 27950 consensus_queue.cc:260] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } }
I20260201 03:03:17.838083 27951 raft_consensus.cc:359] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } }
I20260201 03:03:17.838291 27951 raft_consensus.cc:385] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:17.838330 27779 heartbeater.cc:499] Master 127.18.203.254:44641 was elected leader, sending a full tablet report...
I20260201 03:03:17.838351 27951 raft_consensus.cc:740] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273bae1caac44d809d0ea8a3417710b1, State: Initialized, Role: FOLLOWER
I20260201 03:03:17.838600 27950 ts_tablet_manager.cc:1434] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260201 03:03:17.838728 27637 heartbeater.cc:499] Master 127.18.203.254:44641 was elected leader, sending a full tablet report...
I20260201 03:03:17.839566 27949 ts_tablet_manager.cc:1434] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260201 03:03:17.839660 27951 consensus_queue.cc:260] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } }
I20260201 03:03:17.840399 27951 ts_tablet_manager.cc:1434] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260201 03:03:17.840677 27921 heartbeater.cc:499] Master 127.18.203.254:44641 was elected leader, sending a full tablet report...
W20260201 03:03:18.014397 27922 tablet.cc:2378] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260201 03:03:18.031248 27780 tablet.cc:2378] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260201 03:03:18.051278 27638 tablet.cc:2378] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:03:18.157387 27957 raft_consensus.cc:493] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:03:18.157586 27957 raft_consensus.cc:515] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } }
I20260201 03:03:18.158771 27957 leader_election.cc:290] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02b549b583fb400a89689579b2caa287 (127.18.203.194:41813), 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193:43881)
I20260201 03:03:18.162046 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.159007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:41813 (local address 127.18.203.195:53051)
0201 03:03:18.159242 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:18.159257 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:18.159368 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:18.159660 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:18.159662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:18.159677 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:18.159940 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.159948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.160917 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.160919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:18.161663 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.161671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.161786 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.161809 (+ 23us) client_negotiation.cc:769] Sending connection context
0201 03:03:18.161857 (+ 48us) client_negotiation.cc:240] Negotiation successful
0201 03:03:18.161915 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260201 03:03:18.162388 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.159099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:53051 (local address 127.18.203.194:41813)
0201 03:03:18.159244 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.159249 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.159343 (+ 94us) server_negotiation.cc:408] Connection header received
0201 03:03:18.159486 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.159489 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.159548 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.159651 (+ 103us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:18.160091 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.160811 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.161781 (+ 970us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.162129 (+ 348us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.162169 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.162221 (+ 52us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.162278 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260201 03:03:18.163107 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.159148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:57839 (local address 127.18.203.193:43881)
0201 03:03:18.159278 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.159282 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.159528 (+ 246us) server_negotiation.cc:408] Connection header received
0201 03:03:18.159603 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.159626 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.159675 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.159747 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:18.160129 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.161189 (+ 1060us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.162262 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.162702 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.162739 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.162956 (+ 217us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.163020 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260201 03:03:18.163111 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.159045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:43881 (local address 127.18.203.195:57839)
0201 03:03:18.159445 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:18.159457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:18.159532 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:18.159762 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:18.159765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:18.159779 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:18.160026 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.160033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.161294 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.161298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:18.162125 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.162134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.162807 (+ 673us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.162823 (+ 16us) client_negotiation.cc:769] Sending connection context
0201 03:03:18.162916 (+ 93us) client_negotiation.cc:240] Negotiation successful
0201 03:03:18.162974 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":115,"threads_started":1}
I20260201 03:03:18.163041 27733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f19b65a58cde46e3a92f6713bf6b3a79" candidate_uuid: "273bae1caac44d809d0ea8a3417710b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b549b583fb400a89689579b2caa287" is_pre_election: true
I20260201 03:03:18.163343 27733 raft_consensus.cc:2468] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 273bae1caac44d809d0ea8a3417710b1 in term 0.
I20260201 03:03:18.163781 27591 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f19b65a58cde46e3a92f6713bf6b3a79" candidate_uuid: "273bae1caac44d809d0ea8a3417710b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c779a097be24011a4b5fb6ddf86a532" is_pre_election: true
I20260201 03:03:18.163815 27809 leader_election.cc:304] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b549b583fb400a89689579b2caa287, 273bae1caac44d809d0ea8a3417710b1; no voters:
I20260201 03:03:18.164006 27591 raft_consensus.cc:2468] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 273bae1caac44d809d0ea8a3417710b1 in term 0.
I20260201 03:03:18.164058 27957 raft_consensus.cc:2804] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:03:18.164117 27957 raft_consensus.cc:493] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:03:18.164166 27957 raft_consensus.cc:3060] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.165107 27957 raft_consensus.cc:515] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } }
I20260201 03:03:18.165493 27957 leader_election.cc:290] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [CANDIDATE]: Term 1 election: Requested vote from peers 02b549b583fb400a89689579b2caa287 (127.18.203.194:41813), 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193:43881)
I20260201 03:03:18.165782 27733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f19b65a58cde46e3a92f6713bf6b3a79" candidate_uuid: "273bae1caac44d809d0ea8a3417710b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b549b583fb400a89689579b2caa287"
I20260201 03:03:18.165807 27591 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f19b65a58cde46e3a92f6713bf6b3a79" candidate_uuid: "273bae1caac44d809d0ea8a3417710b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c779a097be24011a4b5fb6ddf86a532"
I20260201 03:03:18.165889 27733 raft_consensus.cc:3060] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.165926 27591 raft_consensus.cc:3060] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.166807 27733 raft_consensus.cc:2468] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 273bae1caac44d809d0ea8a3417710b1 in term 1.
I20260201 03:03:18.166901 27591 raft_consensus.cc:2468] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 273bae1caac44d809d0ea8a3417710b1 in term 1.
I20260201 03:03:18.167111 27809 leader_election.cc:304] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b549b583fb400a89689579b2caa287, 273bae1caac44d809d0ea8a3417710b1; no voters:
I20260201 03:03:18.167321 27957 raft_consensus.cc:2804] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:18.167512 27957 raft_consensus.cc:697] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 1 LEADER]: Becoming Leader. State: Replica: 273bae1caac44d809d0ea8a3417710b1, State: Running, Role: LEADER
I20260201 03:03:18.167850 27957 consensus_queue.cc:237] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } }
I20260201 03:03:18.170820 27431 catalog_manager.cc:5654] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195). New cstate: current_term: 1 leader_uuid: "273bae1caac44d809d0ea8a3417710b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } health_report { overall_health: UNKNOWN } } }
I20260201 03:03:18.268589 27944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.265654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.18.203.195:33317)
0201 03:03:18.265767 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.265771 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.265784 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:18.265933 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.265936 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.266012 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.266105 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:03:18.266436 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.266949 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.267596 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.267779 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.267836 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:03:18.268306 (+ 470us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:03:18.268395 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.268436 (+ 41us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.268481 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260201 03:03:18.273789 27591 raft_consensus.cc:1275] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Refusing update from remote peer 273bae1caac44d809d0ea8a3417710b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:18.273789 27733 raft_consensus.cc:1275] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Refusing update from remote peer 273bae1caac44d809d0ea8a3417710b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:18.274390 27957 consensus_queue.cc:1048] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:18.274561 27959 consensus_queue.cc:1048] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:18.280757 27962 mvcc.cc:204] Tried to move back new op lower bound from 7249571832922787840 to 7249571832498049024. Current Snapshot: MvccSnapshot[applied={T|T < 7249571832922787840}]
I20260201 03:03:18.281265 27963 mvcc.cc:204] Tried to move back new op lower bound from 7249571832922787840 to 7249571832498049024. Current Snapshot: MvccSnapshot[applied={T|T < 7249571832922787840}]
I20260201 03:03:18.282712 27964 mvcc.cc:204] Tried to move back new op lower bound from 7249571832922787840 to 7249571832498049024. Current Snapshot: MvccSnapshot[applied={T|T < 7249571832922787840}]
I20260201 03:03:18.291433 27432 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:18.291658 27432 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:18.294176 27432 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785: Sending DeleteTablet for 3 replicas of tablet f19b65a58cde46e3a92f6713bf6b3a79
I20260201 03:03:18.294929 27713 tablet_service.cc:1552] Processing DeleteTablet for tablet f19b65a58cde46e3a92f6713bf6b3a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:39382
I20260201 03:03:18.294986 27571 tablet_service.cc:1552] Processing DeleteTablet for tablet f19b65a58cde46e3a92f6713bf6b3a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:40060
I20260201 03:03:18.295486 27972 tablet_replica.cc:333] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532: stopping tablet replica
I20260201 03:03:18.295583 27855 tablet_service.cc:1552] Processing DeleteTablet for tablet f19b65a58cde46e3a92f6713bf6b3a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:41858
I20260201 03:03:18.295786 27972 raft_consensus.cc:2243] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:18.295835 27971 tablet_replica.cc:333] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287: stopping tablet replica
I20260201 03:03:18.296139 27972 raft_consensus.cc:2272] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:18.296068 27432 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260201 03:03:18.296128 27971 raft_consensus.cc:2243] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260201 03:03:18.296486 27432 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:03:18.296723 27971 raft_consensus.cc:2272] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:18.297955 27971 ts_tablet_manager.cc:1916] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:18.300552 27971 ts_tablet_manager.cc:1929] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:18.300638 27971 log.cc:1199] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/f19b65a58cde46e3a92f6713bf6b3a79
I20260201 03:03:18.300945 27971 ts_tablet_manager.cc:1950] T f19b65a58cde46e3a92f6713bf6b3a79 P 02b549b583fb400a89689579b2caa287: Deleting consensus metadata
I20260201 03:03:18.301805 27571 tablet_service.cc:1505] Processing CreateTablet for tablet e5f7afc727f24d8697a3661536127c72 (DEFAULT_TABLE table=test-table [id=227b7d92fe0145a29aab0de7dcf08377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:18.302063 27571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5f7afc727f24d8697a3661536127c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:18.302189 27713 tablet_service.cc:1505] Processing CreateTablet for tablet e5f7afc727f24d8697a3661536127c72 (DEFAULT_TABLE table=test-table [id=227b7d92fe0145a29aab0de7dcf08377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:18.302417 27713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5f7afc727f24d8697a3661536127c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:18.304329 27949 tablet_bootstrap.cc:492] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287: Bootstrap starting.
I20260201 03:03:18.304409 27950 tablet_bootstrap.cc:492] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532: Bootstrap starting.
I20260201 03:03:18.305243 27950 tablet_bootstrap.cc:654] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:18.305322 27949 tablet_bootstrap.cc:654] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:18.306236 27855 tablet_service.cc:1505] Processing CreateTablet for tablet e5f7afc727f24d8697a3661536127c72 (DEFAULT_TABLE table=test-table [id=227b7d92fe0145a29aab0de7dcf08377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:18.306504 27855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5f7afc727f24d8697a3661536127c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:18.306907 27950 tablet_bootstrap.cc:492] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532: No bootstrap required, opened a new log
I20260201 03:03:18.306986 27950 ts_tablet_manager.cc:1403] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260201 03:03:18.307389 27950 raft_consensus.cc:359] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.307478 27950 raft_consensus.cc:385] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:18.307507 27950 raft_consensus.cc:740] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c779a097be24011a4b5fb6ddf86a532, State: Initialized, Role: FOLLOWER
I20260201 03:03:18.307647 27950 consensus_queue.cc:260] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.307873 27950 ts_tablet_manager.cc:1434] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:18.308761 27951 tablet_bootstrap.cc:492] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1: Bootstrap starting.
I20260201 03:03:18.308988 27419 catalog_manager.cc:4985] TS 02b549b583fb400a89689579b2caa287 (127.18.203.194:41813): tablet f19b65a58cde46e3a92f6713bf6b3a79 (table test-table [id=8a775af0a4204eab83a2464c662af54a]) successfully deleted
I20260201 03:03:18.309765 27951 tablet_bootstrap.cc:654] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:18.309857 27949 tablet_bootstrap.cc:492] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287: No bootstrap required, opened a new log
I20260201 03:03:18.309948 27949 ts_tablet_manager.cc:1403] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260201 03:03:18.310392 27949 raft_consensus.cc:359] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.310493 27949 raft_consensus.cc:385] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:18.310521 27949 raft_consensus.cc:740] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b549b583fb400a89689579b2caa287, State: Initialized, Role: FOLLOWER
I20260201 03:03:18.310642 27949 consensus_queue.cc:260] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.310889 27949 ts_tablet_manager.cc:1434] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260201 03:03:18.312178 27951 tablet_bootstrap.cc:492] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1: No bootstrap required, opened a new log
I20260201 03:03:18.312263 27951 ts_tablet_manager.cc:1403] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260201 03:03:18.312714 27951 raft_consensus.cc:359] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.312813 27951 raft_consensus.cc:385] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:18.312846 27951 raft_consensus.cc:740] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273bae1caac44d809d0ea8a3417710b1, State: Initialized, Role: FOLLOWER
I20260201 03:03:18.312963 27951 consensus_queue.cc:260] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.313230 27951 ts_tablet_manager.cc:1434] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:18.315698 27972 ts_tablet_manager.cc:1916] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:18.317862 27972 ts_tablet_manager.cc:1929] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:18.317943 27972 log.cc:1199] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/wal/wals/f19b65a58cde46e3a92f6713bf6b3a79
I20260201 03:03:18.318185 27972 ts_tablet_manager.cc:1950] T f19b65a58cde46e3a92f6713bf6b3a79 P 8c779a097be24011a4b5fb6ddf86a532: Deleting consensus metadata
I20260201 03:03:18.319291 27416 catalog_manager.cc:4985] TS 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193:43881): tablet f19b65a58cde46e3a92f6713bf6b3a79 (table test-table [id=8a775af0a4204eab83a2464c662af54a]) successfully deleted
I20260201 03:03:18.319344 27973 tablet_replica.cc:333] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1: stopping tablet replica
I20260201 03:03:18.319635 27973 raft_consensus.cc:2243] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:03:18.320079 27973 raft_consensus.cc:2272] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:18.321171 27973 ts_tablet_manager.cc:1916] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:18.323294 27973 ts_tablet_manager.cc:1929] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:18.323391 27973 log.cc:1199] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/f19b65a58cde46e3a92f6713bf6b3a79
I20260201 03:03:18.323716 27973 ts_tablet_manager.cc:1950] T f19b65a58cde46e3a92f6713bf6b3a79 P 273bae1caac44d809d0ea8a3417710b1: Deleting consensus metadata
I20260201 03:03:18.324743 27416 catalog_manager.cc:4985] TS 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317): tablet f19b65a58cde46e3a92f6713bf6b3a79 (table test-table [id=8a775af0a4204eab83a2464c662af54a]) successfully deleted
I20260201 03:03:18.368304 27956 raft_consensus.cc:493] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:03:18.368479 27956 raft_consensus.cc:515] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.369764 27956 leader_election.cc:290] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193:43881), 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317)
I20260201 03:03:18.373157 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.369981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:43881 (local address 127.18.203.194:48537)
0201 03:03:18.370253 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:18.370269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:18.370392 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:18.370840 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:18.370843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:18.370859 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:18.371150 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.371158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.372053 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.372068 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:18.372742 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.372751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.372860 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.372883 (+ 23us) client_negotiation.cc:769] Sending connection context
0201 03:03:18.372932 (+ 49us) client_negotiation.cc:240] Negotiation successful
0201 03:03:18.372990 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
I20260201 03:03:18.373584 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.370122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:48537 (local address 127.18.203.193:43881)
0201 03:03:18.370256 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.370261 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.370633 (+ 372us) server_negotiation.cc:408] Connection header received
0201 03:03:18.370690 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.370694 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.370747 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.370833 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:18.371266 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.371927 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.372869 (+ 942us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.373304 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.373359 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.373409 (+ 50us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.373478 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260201 03:03:18.373957 27591 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5f7afc727f24d8697a3661536127c72" candidate_uuid: "02b549b583fb400a89689579b2caa287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c779a097be24011a4b5fb6ddf86a532" is_pre_election: true
I20260201 03:03:18.374082 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.370388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:33317 (local address 127.18.203.194:52613)
0201 03:03:18.370631 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:18.370641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:18.370730 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:18.370999 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:18.371002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:18.371036 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:18.371744 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.371754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.372784 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.372787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:18.373707 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.373717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.373827 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.373844 (+ 17us) client_negotiation.cc:769] Sending connection context
0201 03:03:18.373888 (+ 44us) client_negotiation.cc:240] Negotiation successful
0201 03:03:18.373939 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260201 03:03:18.374110 27591 raft_consensus.cc:2468] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b549b583fb400a89689579b2caa287 in term 0.
I20260201 03:03:18.374430 27664 leader_election.cc:304] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b549b583fb400a89689579b2caa287, 8c779a097be24011a4b5fb6ddf86a532; no voters:
I20260201 03:03:18.374552 27944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.370487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:52613 (local address 127.18.203.195:33317)
0201 03:03:18.370603 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.370607 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.370713 (+ 106us) server_negotiation.cc:408] Connection header received
0201 03:03:18.370832 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.370835 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.370886 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.370951 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:18.371885 (+ 934us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.372644 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.373817 (+ 1173us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.374292 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.374328 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.374379 (+ 51us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.374444 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260201 03:03:18.374665 27956 raft_consensus.cc:2804] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:03:18.374723 27956 raft_consensus.cc:493] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:03:18.374766 27956 raft_consensus.cc:3060] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.375950 27956 raft_consensus.cc:515] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.376001 27870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5f7afc727f24d8697a3661536127c72" candidate_uuid: "02b549b583fb400a89689579b2caa287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273bae1caac44d809d0ea8a3417710b1" is_pre_election: true
I20260201 03:03:18.376199 27870 raft_consensus.cc:2468] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b549b583fb400a89689579b2caa287 in term 0.
I20260201 03:03:18.376340 27956 leader_election.cc:290] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [CANDIDATE]: Term 1 election: Requested vote from peers 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193:43881), 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317)
I20260201 03:03:18.376854 27591 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5f7afc727f24d8697a3661536127c72" candidate_uuid: "02b549b583fb400a89689579b2caa287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c779a097be24011a4b5fb6ddf86a532"
I20260201 03:03:18.376996 27591 raft_consensus.cc:3060] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.378191 27591 raft_consensus.cc:2468] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b549b583fb400a89689579b2caa287 in term 1.
I20260201 03:03:18.378456 27664 leader_election.cc:304] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b549b583fb400a89689579b2caa287, 8c779a097be24011a4b5fb6ddf86a532; no voters:
I20260201 03:03:18.378453 27870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5f7afc727f24d8697a3661536127c72" candidate_uuid: "02b549b583fb400a89689579b2caa287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273bae1caac44d809d0ea8a3417710b1"
I20260201 03:03:18.378561 27870 raft_consensus.cc:3060] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.378619 27956 raft_consensus.cc:2804] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:18.378808 27956 raft_consensus.cc:697] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 1 LEADER]: Becoming Leader. State: Replica: 02b549b583fb400a89689579b2caa287, State: Running, Role: LEADER
I20260201 03:03:18.379065 27956 consensus_queue.cc:237] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.379691 27870 raft_consensus.cc:2468] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b549b583fb400a89689579b2caa287 in term 1.
I20260201 03:03:18.381333 27431 catalog_manager.cc:5654] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02b549b583fb400a89689579b2caa287 (127.18.203.194). New cstate: current_term: 1 leader_uuid: "02b549b583fb400a89689579b2caa287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } health_report { overall_health: UNKNOWN } } }
Feb 01 03:03:18 dist-test-slave-gf59 krb5kdc[27384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914996, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.254@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:18.425459 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.417278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:44641 (local address 127.0.0.1:53424)
0201 03:03:18.417685 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:18.417698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:18.417809 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:18.418084 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:18.418087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:18.418481 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:18.418673 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.418679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.419733 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.419736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:18.420116 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.420122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.420208 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.420746 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:18.420760 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:18.422395 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:18.424230 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:18.424237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:18.424242 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:18.424436 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:18.424653 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:18.424655 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:18.424656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:18.424696 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:18.425023 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:18.425028 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:18.425132 (+ 104us) client_negotiation.cc:769] Sending connection context
0201 03:03:18.425216 (+ 84us) client_negotiation.cc:240] Negotiation successful
0201 03:03:18.425319 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":126,"threads_started":1}
I20260201 03:03:18.425587 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.417361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.18.203.254:44641)
0201 03:03:18.417634 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.417639 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.417774 (+ 135us) server_negotiation.cc:408] Connection header received
0201 03:03:18.417896 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.417900 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.417952 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.418101 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:18.418788 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.419638 (+ 850us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.420213 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.420403 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.422518 (+ 2115us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:18.422536 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:18.422540 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:18.422566 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:18.424134 (+ 1568us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:18.424519 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:18.424523 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:18.424525 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:18.424578 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:18.424759 (+ 181us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:18.424761 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:18.424763 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:18.424935 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:18.425034 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.425281 (+ 247us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.425403 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260201 03:03:18.429134 27431 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:53424:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260201 03:03:18.429803 27431 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:03:18.434909 27571 tablet_service.cc:1505] Processing CreateTablet for tablet b06e4eee96b243c2b8498c9923721b09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c7cc599a6ea64b94b42c193c9037ebe5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:03:18.435292 27571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b06e4eee96b243c2b8498c9923721b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:18.435302 27855 tablet_service.cc:1505] Processing CreateTablet for tablet b06e4eee96b243c2b8498c9923721b09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c7cc599a6ea64b94b42c193c9037ebe5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:03:18.435642 27855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b06e4eee96b243c2b8498c9923721b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:18.438103 27950 tablet_bootstrap.cc:492] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532: Bootstrap starting.
I20260201 03:03:18.438339 27713 tablet_service.cc:1505] Processing CreateTablet for tablet b06e4eee96b243c2b8498c9923721b09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c7cc599a6ea64b94b42c193c9037ebe5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260201 03:03:18.438664 27713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b06e4eee96b243c2b8498c9923721b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:18.439164 27950 tablet_bootstrap.cc:654] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:18.440469 27950 tablet_bootstrap.cc:492] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532: No bootstrap required, opened a new log
I20260201 03:03:18.440564 27950 ts_tablet_manager.cc:1403] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260201 03:03:18.440846 27949 tablet_bootstrap.cc:492] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: Bootstrap starting.
I20260201 03:03:18.440963 27951 tablet_bootstrap.cc:492] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1: Bootstrap starting.
I20260201 03:03:18.441066 27950 raft_consensus.cc:359] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.441179 27950 raft_consensus.cc:385] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:18.441212 27950 raft_consensus.cc:740] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c779a097be24011a4b5fb6ddf86a532, State: Initialized, Role: FOLLOWER
I20260201 03:03:18.441368 27950 consensus_queue.cc:260] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.441613 27955 tablet_replica.cc:442] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } }
I20260201 03:03:18.441730 27955 tablet_replica.cc:445] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:03:18.441818 27949 tablet_bootstrap.cc:654] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:18.441833 27951 tablet_bootstrap.cc:654] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:18.442589 27950 ts_tablet_manager.cc:1434] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:18.443529 27951 tablet_bootstrap.cc:492] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1: No bootstrap required, opened a new log
I20260201 03:03:18.443627 27951 ts_tablet_manager.cc:1403] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260201 03:03:18.444065 27951 raft_consensus.cc:359] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.444167 27951 raft_consensus.cc:385] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:18.444196 27951 raft_consensus.cc:740] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273bae1caac44d809d0ea8a3417710b1, State: Initialized, Role: FOLLOWER
I20260201 03:03:18.444308 27951 consensus_queue.cc:260] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.444576 27959 tablet_replica.cc:442] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } }
I20260201 03:03:18.444711 27959 tablet_replica.cc:445] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:03:18.444582 27951 ts_tablet_manager.cc:1434] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260201 03:03:18.445741 27949 tablet_bootstrap.cc:492] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: No bootstrap required, opened a new log
I20260201 03:03:18.445829 27949 ts_tablet_manager.cc:1403] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260201 03:03:18.446292 27949 raft_consensus.cc:359] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.446386 27949 raft_consensus.cc:385] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:18.446408 27949 raft_consensus.cc:740] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b549b583fb400a89689579b2caa287, State: Initialized, Role: FOLLOWER
I20260201 03:03:18.446511 27949 consensus_queue.cc:260] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.446774 27949 ts_tablet_manager.cc:1434] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260201 03:03:18.446967 27956 tablet_replica.cc:442] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } }
I20260201 03:03:18.447073 27956 tablet_replica.cc:445] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:03:18.607883 27956 raft_consensus.cc:493] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:03:18.608032 27956 raft_consensus.cc:515] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.608475 27956 leader_election.cc:290] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193:43881), 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317)
I20260201 03:03:18.608961 27591 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b06e4eee96b243c2b8498c9923721b09" candidate_uuid: "02b549b583fb400a89689579b2caa287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c779a097be24011a4b5fb6ddf86a532" is_pre_election: true
I20260201 03:03:18.609033 27870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b06e4eee96b243c2b8498c9923721b09" candidate_uuid: "02b549b583fb400a89689579b2caa287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273bae1caac44d809d0ea8a3417710b1" is_pre_election: true
I20260201 03:03:18.609105 27591 raft_consensus.cc:2468] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b549b583fb400a89689579b2caa287 in term 0.
I20260201 03:03:18.609197 27870 raft_consensus.cc:2468] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b549b583fb400a89689579b2caa287 in term 0.
I20260201 03:03:18.609404 27664 leader_election.cc:304] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b549b583fb400a89689579b2caa287, 8c779a097be24011a4b5fb6ddf86a532; no voters:
I20260201 03:03:18.609593 27956 raft_consensus.cc:2804] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:03:18.609655 27956 raft_consensus.cc:493] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:03:18.609704 27956 raft_consensus.cc:3060] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.610587 27956 raft_consensus.cc:515] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.610909 27956 leader_election.cc:290] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [CANDIDATE]: Term 1 election: Requested vote from peers 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193:43881), 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317)
I20260201 03:03:18.611327 27591 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b06e4eee96b243c2b8498c9923721b09" candidate_uuid: "02b549b583fb400a89689579b2caa287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c779a097be24011a4b5fb6ddf86a532"
I20260201 03:03:18.611364 27870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b06e4eee96b243c2b8498c9923721b09" candidate_uuid: "02b549b583fb400a89689579b2caa287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273bae1caac44d809d0ea8a3417710b1"
I20260201 03:03:18.611442 27591 raft_consensus.cc:3060] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.611474 27870 raft_consensus.cc:3060] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.612334 27870 raft_consensus.cc:2468] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b549b583fb400a89689579b2caa287 in term 1.
I20260201 03:03:18.612340 27591 raft_consensus.cc:2468] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b549b583fb400a89689579b2caa287 in term 1.
I20260201 03:03:18.612617 27664 leader_election.cc:304] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b549b583fb400a89689579b2caa287, 273bae1caac44d809d0ea8a3417710b1; no voters:
I20260201 03:03:18.612782 27956 raft_consensus.cc:2804] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:18.612874 27956 raft_consensus.cc:697] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [term 1 LEADER]: Becoming Leader. State: Replica: 02b549b583fb400a89689579b2caa287, State: Running, Role: LEADER
I20260201 03:03:18.613018 27956 consensus_queue.cc:237] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.613724 27979 tablet_replica.cc:442] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: TxnStatusTablet state changed. Reason: New leader 02b549b583fb400a89689579b2caa287. Latest consensus state: current_term: 1 leader_uuid: "02b549b583fb400a89689579b2caa287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } }
I20260201 03:03:18.613807 27979 tablet_replica.cc:445] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:03:18.614063 27992 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260201 03:03:18.614199 27992 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260201 03:03:18.614876 27431 catalog_manager.cc:5654] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02b549b583fb400a89689579b2caa287 (127.18.203.194). New cstate: current_term: 1 leader_uuid: "02b549b583fb400a89689579b2caa287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } health_report { overall_health: UNKNOWN } } }
I20260201 03:03:18.636562 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.633177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:41813 (local address 127.0.0.1:39392)
0201 03:03:18.633424 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:18.633439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:18.633582 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:18.633885 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:18.633888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:18.633896 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:03:18.634101 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.634107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.634797 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.634800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:18.635214 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.635221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.635332 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.635385 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:03:18.636248 (+ 863us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:03:18.636266 (+ 18us) client_negotiation.cc:769] Sending connection context
0201 03:03:18.636335 (+ 69us) client_negotiation.cc:240] Negotiation successful
0201 03:03:18.636418 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260201 03:03:18.636562 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.633285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39392 (local address 127.18.203.194:41813)
0201 03:03:18.633419 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.633424 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.633499 (+ 75us) server_negotiation.cc:408] Connection header received
0201 03:03:18.633713 (+ 214us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.633716 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.633777 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.633862 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:03:18.634208 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.634688 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.635323 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.635505 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.635557 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:03:18.636152 (+ 595us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:03:18.636261 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.636357 (+ 96us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.636415 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260201 03:03:18.639644 27591 raft_consensus.cc:1275] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Refusing update from remote peer 02b549b583fb400a89689579b2caa287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:18.639745 27870 raft_consensus.cc:1275] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Refusing update from remote peer 02b549b583fb400a89689579b2caa287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:18.640169 27979 consensus_queue.cc:1048] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:18.640353 27956 consensus_queue.cc:1048] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:18.641916 27955 tablet_replica.cc:442] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532: TxnStatusTablet state changed. Reason: New leader 02b549b583fb400a89689579b2caa287. Latest consensus state: current_term: 1 leader_uuid: "02b549b583fb400a89689579b2caa287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } }
I20260201 03:03:18.642030 27955 tablet_replica.cc:445] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:03:18.643433 27959 tablet_replica.cc:442] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1: TxnStatusTablet state changed. Reason: New leader 02b549b583fb400a89689579b2caa287. Latest consensus state: current_term: 1 leader_uuid: "02b549b583fb400a89689579b2caa287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } }
I20260201 03:03:18.643546 27959 tablet_replica.cc:445] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:03:18.644699 27979 tablet_replica.cc:442] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02b549b583fb400a89689579b2caa287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } }
I20260201 03:03:18.644793 27979 tablet_replica.cc:445] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:03:18.646847 27956 tablet_replica.cc:442] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02b549b583fb400a89689579b2caa287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } }
I20260201 03:03:18.646948 27956 tablet_replica.cc:445] T b06e4eee96b243c2b8498c9923721b09 P 02b549b583fb400a89689579b2caa287: This TxnStatusTablet replica's current role is: LEADER
I20260201 03:03:18.647130 27959 tablet_replica.cc:442] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02b549b583fb400a89689579b2caa287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } }
I20260201 03:03:18.647231 27959 tablet_replica.cc:445] T b06e4eee96b243c2b8498c9923721b09 P 273bae1caac44d809d0ea8a3417710b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:03:18.647832 27955 tablet_replica.cc:442] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02b549b583fb400a89689579b2caa287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } } }
I20260201 03:03:18.647936 27955 tablet_replica.cc:445] T b06e4eee96b243c2b8498c9923721b09 P 8c779a097be24011a4b5fb6ddf86a532: This TxnStatusTablet replica's current role is: FOLLOWER
I20260201 03:03:18.653347 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.650605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.18.203.194:41813)
0201 03:03:18.650710 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.650713 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.650767 (+ 54us) server_negotiation.cc:408] Connection header received
0201 03:03:18.650886 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.650889 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.650927 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.650986 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:03:18.651369 (+ 383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.651880 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.652736 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.652873 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.652912 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:03:18.652973 (+ 61us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:03:18.653032 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.653160 (+ 128us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.653206 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260201 03:03:18.663322 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.660417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:44113 (local address 127.18.203.194:41813)
0201 03:03:18.660539 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.660544 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.660732 (+ 188us) server_negotiation.cc:408] Connection header received
0201 03:03:18.660861 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.660863 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.660904 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.660964 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:03:18.661402 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.661854 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.662722 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.662863 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.662900 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:03:18.662966 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:03:18.663022 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.663125 (+ 103us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.663165 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260201 03:03:18.663345 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.660354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:41813 (local address 127.18.203.194:44113)
0201 03:03:18.660648 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:18.660663 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:18.660786 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:18.660983 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:18.660987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:18.660994 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:03:18.661313 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.661320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.661955 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.661959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:18.662618 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.662628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.662742 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.662791 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:03:18.663038 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:03:18.663044 (+ 6us) client_negotiation.cc:769] Sending connection context
0201 03:03:18.663114 (+ 70us) client_negotiation.cc:240] Negotiation successful
0201 03:03:18.663180 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":112,"threads_started":1}
I20260201 03:03:18.670388 27870 raft_consensus.cc:1275] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Refusing update from remote peer 02b549b583fb400a89689579b2caa287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:18.670429 27591 raft_consensus.cc:1275] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Refusing update from remote peer 02b549b583fb400a89689579b2caa287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:18.670838 27979 consensus_queue.cc:1048] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:18.671139 27956 consensus_queue.cc:1048] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:18.710853 27432 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:18.711017 27432 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:18.713165 27432 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785: Sending DeleteTablet for 3 replicas of tablet e5f7afc727f24d8697a3661536127c72
I20260201 03:03:18.713909 27571 tablet_service.cc:1552] Processing DeleteTablet for tablet e5f7afc727f24d8697a3661536127c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:40060
I20260201 03:03:18.713933 27855 tablet_service.cc:1552] Processing DeleteTablet for tablet e5f7afc727f24d8697a3661536127c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:41858
I20260201 03:03:18.714164 27972 tablet_replica.cc:333] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532: stopping tablet replica
I20260201 03:03:18.714252 27973 tablet_replica.cc:333] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1: stopping tablet replica
I20260201 03:03:18.714344 27972 raft_consensus.cc:2243] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:18.714368 27973 raft_consensus.cc:2243] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:18.714468 27972 raft_consensus.cc:2272] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:18.714468 27973 raft_consensus.cc:2272] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:18.714915 27712 tablet_service.cc:1552] Processing DeleteTablet for tablet e5f7afc727f24d8697a3661536127c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:39382
I20260201 03:03:18.715209 27972 ts_tablet_manager.cc:1916] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:18.715719 27971 tablet_replica.cc:333] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287: stopping tablet replica
I20260201 03:03:18.715837 27971 raft_consensus.cc:2243] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:03:18.716045 27973 ts_tablet_manager.cc:1916] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:18.716161 27971 raft_consensus.cc:2272] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:18.718699 27973 ts_tablet_manager.cc:1929] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:03:18.718699 27972 ts_tablet_manager.cc:1929] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:03:18.718780 27973 log.cc:1199] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/e5f7afc727f24d8697a3661536127c72
I20260201 03:03:18.718780 27972 log.cc:1199] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/wal/wals/e5f7afc727f24d8697a3661536127c72
I20260201 03:03:18.719023 27973 ts_tablet_manager.cc:1950] T e5f7afc727f24d8697a3661536127c72 P 273bae1caac44d809d0ea8a3417710b1: Deleting consensus metadata
I20260201 03:03:18.719023 27972 ts_tablet_manager.cc:1950] T e5f7afc727f24d8697a3661536127c72 P 8c779a097be24011a4b5fb6ddf86a532: Deleting consensus metadata
I20260201 03:03:18.719990 27416 catalog_manager.cc:4985] TS 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193:43881): tablet e5f7afc727f24d8697a3661536127c72 (table test-table [id=227b7d92fe0145a29aab0de7dcf08377]) successfully deleted
I20260201 03:03:18.720227 27971 ts_tablet_manager.cc:1916] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:18.720237 27416 catalog_manager.cc:4985] TS 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317): tablet e5f7afc727f24d8697a3661536127c72 (table test-table [id=227b7d92fe0145a29aab0de7dcf08377]) successfully deleted
I20260201 03:03:18.722270 27971 ts_tablet_manager.cc:1929] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:03:18.722339 27971 log.cc:1199] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/e5f7afc727f24d8697a3661536127c72
I20260201 03:03:18.722512 27971 ts_tablet_manager.cc:1950] T e5f7afc727f24d8697a3661536127c72 P 02b549b583fb400a89689579b2caa287: Deleting consensus metadata
I20260201 03:03:18.723321 27419 catalog_manager.cc:4985] TS 02b549b583fb400a89689579b2caa287 (127.18.203.194:41813): tablet e5f7afc727f24d8697a3661536127c72 (table test-table [id=227b7d92fe0145a29aab0de7dcf08377]) successfully deleted
Feb 01 03:03:18 dist-test-slave-gf59 krb5kdc[27384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 01 03:03:18 dist-test-slave-gf59 krb5kdc[27384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:18 dist-test-slave-gf59 krb5kdc[27384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
W20260201 03:03:18.748085 27987 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:18.741855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.18.203.254:44641)
0201 03:03:18.741995 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.742000 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.742177 (+ 177us) server_negotiation.cc:408] Connection header received
0201 03:03:18.742317 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.742321 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.742373 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.742444 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:18.743320 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.743920 (+ 600us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.744577 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.744770 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.747650 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53436: BlockingRecv error: recv got EOF from 127.0.0.1:53436 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Feb 01 03:03:18 dist-test-slave-gf59 krb5kdc[27384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769914998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM
I20260201 03:03:18.756539 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.749288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.18.203.254:44641)
0201 03:03:18.749407 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.749412 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.749531 (+ 119us) server_negotiation.cc:408] Connection header received
0201 03:03:18.749648 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.749651 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.749694 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.749759 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:18.750417 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.750883 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.751465 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.751668 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.753810 (+ 2142us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:18.753827 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:18.753830 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:18.753859 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:18.755118 (+ 1259us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:18.755515 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:18.755518 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:18.755520 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:18.755566 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:18.755781 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:18.755784 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:18.755787 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:18.755936 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:18.756074 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.756257 (+ 183us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.756381 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260201 03:03:18.758937 27432 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:03:18.759397 27432 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:03:18.764052 27571 tablet_service.cc:1505] Processing CreateTablet for tablet 0de736d43d7644da8ad843dd56d89433 (DEFAULT_TABLE table=test-table [id=ba9084b32d0f4ab7ba4e4d366bbc5e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:18.764353 27571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de736d43d7644da8ad843dd56d89433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:18.766395 27950 tablet_bootstrap.cc:492] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532: Bootstrap starting.
I20260201 03:03:18.767268 27950 tablet_bootstrap.cc:654] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:18.768437 27855 tablet_service.cc:1505] Processing CreateTablet for tablet 0de736d43d7644da8ad843dd56d89433 (DEFAULT_TABLE table=test-table [id=ba9084b32d0f4ab7ba4e4d366bbc5e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:18.768709 27855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de736d43d7644da8ad843dd56d89433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:18.768714 27950 tablet_bootstrap.cc:492] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532: No bootstrap required, opened a new log
I20260201 03:03:18.768784 27950 ts_tablet_manager.cc:1403] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:18.768720 27712 tablet_service.cc:1505] Processing CreateTablet for tablet 0de736d43d7644da8ad843dd56d89433 (DEFAULT_TABLE table=test-table [id=ba9084b32d0f4ab7ba4e4d366bbc5e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:18.769011 27712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de736d43d7644da8ad843dd56d89433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:18.769254 27950 raft_consensus.cc:359] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.769353 27950 raft_consensus.cc:385] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:18.769380 27950 raft_consensus.cc:740] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c779a097be24011a4b5fb6ddf86a532, State: Initialized, Role: FOLLOWER
I20260201 03:03:18.769495 27950 consensus_queue.cc:260] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.769735 27950 ts_tablet_manager.cc:1434] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:18.771047 27951 tablet_bootstrap.cc:492] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1: Bootstrap starting.
I20260201 03:03:18.771353 27949 tablet_bootstrap.cc:492] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287: Bootstrap starting.
I20260201 03:03:18.772023 27951 tablet_bootstrap.cc:654] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:18.772336 27949 tablet_bootstrap.cc:654] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:18.773056 27951 tablet_bootstrap.cc:492] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1: No bootstrap required, opened a new log
I20260201 03:03:18.773147 27951 ts_tablet_manager.cc:1403] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260201 03:03:18.773502 27949 tablet_bootstrap.cc:492] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287: No bootstrap required, opened a new log
I20260201 03:03:18.773494 27951 raft_consensus.cc:359] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.773571 27951 raft_consensus.cc:385] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:18.773578 27949 ts_tablet_manager.cc:1403] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:18.773590 27951 raft_consensus.cc:740] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273bae1caac44d809d0ea8a3417710b1, State: Initialized, Role: FOLLOWER
I20260201 03:03:18.773696 27951 consensus_queue.cc:260] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.773931 27951 ts_tablet_manager.cc:1434] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260201 03:03:18.774015 27949 raft_consensus.cc:359] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.774130 27949 raft_consensus.cc:385] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:18.774160 27949 raft_consensus.cc:740] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b549b583fb400a89689579b2caa287, State: Initialized, Role: FOLLOWER
I20260201 03:03:18.774284 27949 consensus_queue.cc:260] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.774520 27949 ts_tablet_manager.cc:1434] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:18.792104 27955 raft_consensus.cc:493] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:03:18.792237 27955 raft_consensus.cc:515] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.793277 27955 leader_election.cc:290] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02b549b583fb400a89689579b2caa287 (127.18.203.194:41813), 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317)
I20260201 03:03:18.796973 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.793451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:41813 (local address 127.18.203.193:49547)
0201 03:03:18.794128 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:18.794143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:18.794287 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:18.794668 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:18.794671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:18.794691 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:18.795169 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.795176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.795907 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.795910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:18.796542 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.796553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.796688 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.796714 (+ 26us) client_negotiation.cc:769] Sending connection context
0201 03:03:18.796764 (+ 50us) client_negotiation.cc:240] Negotiation successful
0201 03:03:18.796825 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":75,"threads_started":1}
I20260201 03:03:18.797410 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.793772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:49547 (local address 127.18.203.194:41813)
0201 03:03:18.793896 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.793901 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.794207 (+ 306us) server_negotiation.cc:408] Connection header received
0201 03:03:18.794413 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.794417 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.794468 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.794517 (+ 49us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:18.795283 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.795805 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.796652 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.797145 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.797183 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.797234 (+ 51us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.797293 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260201 03:03:18.797873 27733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0de736d43d7644da8ad843dd56d89433" candidate_uuid: "8c779a097be24011a4b5fb6ddf86a532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b549b583fb400a89689579b2caa287" is_pre_election: true
I20260201 03:03:18.798048 27733 raft_consensus.cc:2468] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c779a097be24011a4b5fb6ddf86a532 in term 0.
I20260201 03:03:18.798453 27525 leader_election.cc:304] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b549b583fb400a89689579b2caa287, 8c779a097be24011a4b5fb6ddf86a532; no voters:
I20260201 03:03:18.798735 27955 raft_consensus.cc:2804] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:03:18.798803 27955 raft_consensus.cc:493] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:03:18.798841 27955 raft_consensus.cc:3060] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.799836 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.793541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:33317 (local address 127.18.203.193:43705)
0201 03:03:18.796351 (+ 2810us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:18.796367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:18.796490 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:18.796807 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:18.796810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:18.796827 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:18.797097 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.797106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.797958 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.797961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:18.798795 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.798805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.799511 (+ 706us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.799528 (+ 17us) client_negotiation.cc:769] Sending connection context
0201 03:03:18.799603 (+ 75us) client_negotiation.cc:240] Negotiation successful
0201 03:03:18.799692 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2731,"thread_start_us":90,"threads_started":1}
I20260201 03:03:18.799892 27944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.793841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:43705 (local address 127.18.203.195:33317)
0201 03:03:18.793944 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.793948 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.796457 (+ 2509us) server_negotiation.cc:408] Connection header received
0201 03:03:18.796590 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.796593 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.796649 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.796714 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:18.797216 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.797849 (+ 633us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.798930 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.799387 (+ 457us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.799424 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.799701 (+ 277us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.799781 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260201 03:03:18.800099 27955 raft_consensus.cc:515] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.800318 27870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0de736d43d7644da8ad843dd56d89433" candidate_uuid: "8c779a097be24011a4b5fb6ddf86a532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273bae1caac44d809d0ea8a3417710b1" is_pre_election: true
I20260201 03:03:18.800436 27870 raft_consensus.cc:2468] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c779a097be24011a4b5fb6ddf86a532 in term 0.
I20260201 03:03:18.800483 27955 leader_election.cc:290] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [CANDIDATE]: Term 1 election: Requested vote from peers 02b549b583fb400a89689579b2caa287 (127.18.203.194:41813), 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317)
I20260201 03:03:18.800844 27870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0de736d43d7644da8ad843dd56d89433" candidate_uuid: "8c779a097be24011a4b5fb6ddf86a532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273bae1caac44d809d0ea8a3417710b1"
I20260201 03:03:18.800947 27870 raft_consensus.cc:3060] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.801218 27733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0de736d43d7644da8ad843dd56d89433" candidate_uuid: "8c779a097be24011a4b5fb6ddf86a532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b549b583fb400a89689579b2caa287"
I20260201 03:03:18.801333 27733 raft_consensus.cc:3060] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.802126 27870 raft_consensus.cc:2468] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c779a097be24011a4b5fb6ddf86a532 in term 1.
I20260201 03:03:18.802479 27522 leader_election.cc:304] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 273bae1caac44d809d0ea8a3417710b1, 8c779a097be24011a4b5fb6ddf86a532; no voters:
I20260201 03:03:18.802649 27733 raft_consensus.cc:2468] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c779a097be24011a4b5fb6ddf86a532 in term 1.
I20260201 03:03:18.802702 27955 raft_consensus.cc:2804] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:18.802912 27955 raft_consensus.cc:697] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 LEADER]: Becoming Leader. State: Replica: 8c779a097be24011a4b5fb6ddf86a532, State: Running, Role: LEADER
I20260201 03:03:18.803134 27955 consensus_queue.cc:237] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.805215 27431 catalog_manager.cc:5654] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193). New cstate: current_term: 1 leader_uuid: "8c779a097be24011a4b5fb6ddf86a532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } health_report { overall_health: UNKNOWN } } }
I20260201 03:03:18.817965 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.815117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40064 (local address 127.18.203.193:43881)
0201 03:03:18.815255 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.815260 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.815274 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:03:18.815347 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.815350 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.815401 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.815482 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:03:18.815858 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.816322 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.816939 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.817112 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.817165 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:03:18.817609 (+ 444us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:03:18.817710 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.817788 (+ 78us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.817835 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260201 03:03:18.820425 27733 raft_consensus.cc:1275] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Refusing update from remote peer 8c779a097be24011a4b5fb6ddf86a532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:18.820537 27870 raft_consensus.cc:1275] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Refusing update from remote peer 8c779a097be24011a4b5fb6ddf86a532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:18.820942 27955 consensus_queue.cc:1048] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:18.821116 28021 consensus_queue.cc:1048] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:18.831758 27432 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:18.831924 27432 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:18.834854 27432 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785: Sending DeleteTablet for 3 replicas of tablet 0de736d43d7644da8ad843dd56d89433
I20260201 03:03:18.835729 27571 tablet_service.cc:1552] Processing DeleteTablet for tablet 0de736d43d7644da8ad843dd56d89433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:40060
I20260201 03:03:18.835927 27855 tablet_service.cc:1552] Processing DeleteTablet for tablet 0de736d43d7644da8ad843dd56d89433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:41858
I20260201 03:03:18.836113 27972 tablet_replica.cc:333] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532: stopping tablet replica
I20260201 03:03:18.836229 27972 raft_consensus.cc:2243] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:03:18.836527 27972 raft_consensus.cc:2272] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:18.836709 27973 tablet_replica.cc:333] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1: stopping tablet replica
I20260201 03:03:18.836817 27973 raft_consensus.cc:2243] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:18.836851 27712 tablet_service.cc:1552] Processing DeleteTablet for tablet 0de736d43d7644da8ad843dd56d89433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:39382
I20260201 03:03:18.836925 27973 raft_consensus.cc:2272] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:18.837041 27971 tablet_replica.cc:333] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287: stopping tablet replica
I20260201 03:03:18.837146 27971 raft_consensus.cc:2243] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:18.837206 27713 tablet_service.cc:1552] Processing DeleteTablet for tablet 0de736d43d7644da8ad843dd56d89433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:39382
I20260201 03:03:18.837287 27971 raft_consensus.cc:2272] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:18.838158 27971 ts_tablet_manager.cc:1916] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:18.838160 27973 ts_tablet_manager.cc:1916] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:18.840210 27972 ts_tablet_manager.cc:1916] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:18.840577 27971 ts_tablet_manager.cc:1929] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:18.840647 27971 log.cc:1199] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/0de736d43d7644da8ad843dd56d89433
I20260201 03:03:18.840912 27971 ts_tablet_manager.cc:1950] T 0de736d43d7644da8ad843dd56d89433 P 02b549b583fb400a89689579b2caa287: Deleting consensus metadata
I20260201 03:03:18.841966 27419 catalog_manager.cc:4985] TS 02b549b583fb400a89689579b2caa287 (127.18.203.194:41813): tablet 0de736d43d7644da8ad843dd56d89433 (table test-table [id=ba9084b32d0f4ab7ba4e4d366bbc5e0c]) successfully deleted
I20260201 03:03:18.842376 27972 ts_tablet_manager.cc:1929] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:18.842451 27972 log.cc:1199] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-0/wal/wals/0de736d43d7644da8ad843dd56d89433
I20260201 03:03:18.842693 27972 ts_tablet_manager.cc:1950] T 0de736d43d7644da8ad843dd56d89433 P 8c779a097be24011a4b5fb6ddf86a532: Deleting consensus metadata
I20260201 03:03:18.843204 27432 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:03:18.843756 27432 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:03:18.844340 27973 ts_tablet_manager.cc:1929] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:18.844409 27973 log.cc:1199] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/0de736d43d7644da8ad843dd56d89433
I20260201 03:03:18.844641 27973 ts_tablet_manager.cc:1950] T 0de736d43d7644da8ad843dd56d89433 P 273bae1caac44d809d0ea8a3417710b1: Deleting consensus metadata
I20260201 03:03:18.844722 27416 catalog_manager.cc:4985] TS 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193:43881): tablet 0de736d43d7644da8ad843dd56d89433 (table test-table [id=ba9084b32d0f4ab7ba4e4d366bbc5e0c]) successfully deleted
I20260201 03:03:18.845357 27416 catalog_manager.cc:4985] TS 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317): tablet 0de736d43d7644da8ad843dd56d89433 (table test-table [id=ba9084b32d0f4ab7ba4e4d366bbc5e0c]) successfully deleted
W20260201 03:03:18.845957 27419 catalog_manager.cc:4952] TS 02b549b583fb400a89689579b2caa287 (127.18.203.194:41813): delete failed for tablet 0de736d43d7644da8ad843dd56d89433 because the tablet was not found. No further retry: Not found: Tablet not found: 0de736d43d7644da8ad843dd56d89433
I20260201 03:03:18.851184 27571 tablet_service.cc:1505] Processing CreateTablet for tablet 04462ede3f5c45938383ef31924d2f87 (DEFAULT_TABLE table=test-table [id=c6d41b028a8749a4b4e8c5f4768561a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:18.851491 27571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04462ede3f5c45938383ef31924d2f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:18.851428 27855 tablet_service.cc:1505] Processing CreateTablet for tablet 04462ede3f5c45938383ef31924d2f87 (DEFAULT_TABLE table=test-table [id=c6d41b028a8749a4b4e8c5f4768561a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:18.851722 27855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04462ede3f5c45938383ef31924d2f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:18.853688 27950 tablet_bootstrap.cc:492] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532: Bootstrap starting.
I20260201 03:03:18.853849 27951 tablet_bootstrap.cc:492] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1: Bootstrap starting.
I20260201 03:03:18.854671 27951 tablet_bootstrap.cc:654] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:18.854671 27950 tablet_bootstrap.cc:654] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:18.855782 27713 tablet_service.cc:1505] Processing CreateTablet for tablet 04462ede3f5c45938383ef31924d2f87 (DEFAULT_TABLE table=test-table [id=c6d41b028a8749a4b4e8c5f4768561a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:18.856043 27713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04462ede3f5c45938383ef31924d2f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:18.857195 27951 tablet_bootstrap.cc:492] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1: No bootstrap required, opened a new log
I20260201 03:03:18.857290 27951 ts_tablet_manager.cc:1403] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260201 03:03:18.857679 27951 raft_consensus.cc:359] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.857777 27951 raft_consensus.cc:385] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:18.857806 27951 raft_consensus.cc:740] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273bae1caac44d809d0ea8a3417710b1, State: Initialized, Role: FOLLOWER
I20260201 03:03:18.857923 27951 consensus_queue.cc:260] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.858182 27951 ts_tablet_manager.cc:1434] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:18.858479 27949 tablet_bootstrap.cc:492] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287: Bootstrap starting.
I20260201 03:03:18.859368 27949 tablet_bootstrap.cc:654] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:18.861197 27949 tablet_bootstrap.cc:492] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287: No bootstrap required, opened a new log
I20260201 03:03:18.861292 27949 ts_tablet_manager.cc:1403] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260201 03:03:18.861692 27949 raft_consensus.cc:359] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.861788 27949 raft_consensus.cc:385] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:18.861816 27949 raft_consensus.cc:740] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b549b583fb400a89689579b2caa287, State: Initialized, Role: FOLLOWER
I20260201 03:03:18.861929 27949 consensus_queue.cc:260] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.862174 27949 ts_tablet_manager.cc:1434] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260201 03:03:18.872176 27950 tablet_bootstrap.cc:492] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532: No bootstrap required, opened a new log
I20260201 03:03:18.872321 27950 ts_tablet_manager.cc:1403] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20260201 03:03:18.872862 27950 raft_consensus.cc:359] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.873000 27950 raft_consensus.cc:385] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:18.873042 27950 raft_consensus.cc:740] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c779a097be24011a4b5fb6ddf86a532, State: Initialized, Role: FOLLOWER
I20260201 03:03:18.873170 27950 consensus_queue.cc:260] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.873466 27950 ts_tablet_manager.cc:1434] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260201 03:03:18.877363 28026 raft_consensus.cc:493] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:03:18.877473 28026 raft_consensus.cc:515] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.877899 28026 leader_election.cc:290] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02b549b583fb400a89689579b2caa287 (127.18.203.194:41813), 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317)
I20260201 03:03:18.878345 27870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04462ede3f5c45938383ef31924d2f87" candidate_uuid: "8c779a097be24011a4b5fb6ddf86a532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273bae1caac44d809d0ea8a3417710b1" is_pre_election: true
I20260201 03:03:18.878365 27733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04462ede3f5c45938383ef31924d2f87" candidate_uuid: "8c779a097be24011a4b5fb6ddf86a532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b549b583fb400a89689579b2caa287" is_pre_election: true
I20260201 03:03:18.878489 27870 raft_consensus.cc:2468] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c779a097be24011a4b5fb6ddf86a532 in term 0.
I20260201 03:03:18.878511 27733 raft_consensus.cc:2468] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c779a097be24011a4b5fb6ddf86a532 in term 0.
I20260201 03:03:18.878772 27522 leader_election.cc:304] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 273bae1caac44d809d0ea8a3417710b1, 8c779a097be24011a4b5fb6ddf86a532; no voters:
I20260201 03:03:18.878930 28026 raft_consensus.cc:2804] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:03:18.878969 28026 raft_consensus.cc:493] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:03:18.879009 28026 raft_consensus.cc:3060] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.880278 28026 raft_consensus.cc:515] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.880915 27733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04462ede3f5c45938383ef31924d2f87" candidate_uuid: "8c779a097be24011a4b5fb6ddf86a532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b549b583fb400a89689579b2caa287"
I20260201 03:03:18.880937 27870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04462ede3f5c45938383ef31924d2f87" candidate_uuid: "8c779a097be24011a4b5fb6ddf86a532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273bae1caac44d809d0ea8a3417710b1"
I20260201 03:03:18.881024 27733 raft_consensus.cc:3060] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.881047 27870 raft_consensus.cc:3060] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:18.882081 27870 raft_consensus.cc:2468] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c779a097be24011a4b5fb6ddf86a532 in term 1.
I20260201 03:03:18.882084 27733 raft_consensus.cc:2468] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c779a097be24011a4b5fb6ddf86a532 in term 1.
I20260201 03:03:18.882408 27522 leader_election.cc:304] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 273bae1caac44d809d0ea8a3417710b1, 8c779a097be24011a4b5fb6ddf86a532; no voters:
I20260201 03:03:18.882586 28027 raft_consensus.cc:2804] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:18.882660 28027 raft_consensus.cc:697] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 LEADER]: Becoming Leader. State: Replica: 8c779a097be24011a4b5fb6ddf86a532, State: Running, Role: LEADER
I20260201 03:03:18.882787 28027 consensus_queue.cc:237] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } }
I20260201 03:03:18.884527 27432 catalog_manager.cc:5654] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193). New cstate: current_term: 1 leader_uuid: "8c779a097be24011a4b5fb6ddf86a532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c779a097be24011a4b5fb6ddf86a532" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 43881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 } health_report { overall_health: UNKNOWN } } }
I20260201 03:03:18.885334 28026 leader_election.cc:290] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [CANDIDATE]: Term 1 election: Requested vote from peers 02b549b583fb400a89689579b2caa287 (127.18.203.194:41813), 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317)
I20260201 03:03:18.913061 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.909450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:41813 (local address 127.18.203.193:32997)
0201 03:03:18.909745 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:18.909762 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:18.909859 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:18.910161 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:18.910164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:18.910172 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:03:18.910505 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.910514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.911512 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.911516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:18.912229 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.912239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.912358 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.912404 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:03:18.912763 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:03:18.912768 (+ 5us) client_negotiation.cc:769] Sending connection context
0201 03:03:18.912828 (+ 60us) client_negotiation.cc:240] Negotiation successful
0201 03:03:18.912896 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
I20260201 03:03:18.913040 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.909822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:32997 (local address 127.18.203.194:41813)
0201 03:03:18.909937 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.909941 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.909954 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:18.910002 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.910005 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.910059 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.910134 (+ 75us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:03:18.910618 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.911378 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.912353 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.912547 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.912593 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:03:18.912674 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:03:18.912753 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.912848 (+ 95us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.912896 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260201 03:03:18.919972 27733 raft_consensus.cc:1275] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Refusing update from remote peer 8c779a097be24011a4b5fb6ddf86a532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:18.920348 27875 raft_consensus.cc:1275] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Refusing update from remote peer 8c779a097be24011a4b5fb6ddf86a532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:18.920743 28026 consensus_queue.cc:1048] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b549b583fb400a89689579b2caa287" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 41813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:18.921631 28026 consensus_queue.cc:1048] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "273bae1caac44d809d0ea8a3417710b1" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 33317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:18.946882 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.943404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:43881 (local address 127.18.203.194:35409)
0201 03:03:18.943683 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:18.943698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:18.943776 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:18.944326 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:18.944329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:18.944339 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0201 03:03:18.944570 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.944578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.945581 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.945586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:18.946070 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:18.946079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.946185 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.946216 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0201 03:03:18.946599 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0201 03:03:18.946604 (+ 5us) client_negotiation.cc:769] Sending connection context
0201 03:03:18.946660 (+ 56us) client_negotiation.cc:240] Negotiation successful
0201 03:03:18.946731 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260201 03:03:18.947880 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.943907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:35409 (local address 127.18.203.193:43881)
0201 03:03:18.944018 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:18.944022 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:18.944034 (+ 12us) server_negotiation.cc:408] Connection header received
0201 03:03:18.944073 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:18.944076 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:18.944127 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:18.944199 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:03:18.944683 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.945476 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:18.946180 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:18.946367 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:18.946413 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:03:18.946496 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:03:18.946561 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:18.947704 (+ 1143us) server_negotiation.cc:300] Negotiation successful
0201 03:03:18.947757 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260201 03:03:18.969913 27430 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:18.970103 27430 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:18.972388 27430 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b606226d8d014ecbbb0f1f1d27f7b785: Sending DeleteTablet for 3 replicas of tablet 04462ede3f5c45938383ef31924d2f87
I20260201 03:03:18.973016 27855 tablet_service.cc:1552] Processing DeleteTablet for tablet 04462ede3f5c45938383ef31924d2f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:41858
I20260201 03:03:18.973276 27973 tablet_replica.cc:333] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1: stopping tablet replica
I20260201 03:03:18.973417 27973 raft_consensus.cc:2243] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:18.973543 27973 raft_consensus.cc:2272] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:18.973537 27571 tablet_service.cc:1552] Processing DeleteTablet for tablet 04462ede3f5c45938383ef31924d2f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:40060
I20260201 03:03:18.973814 27972 tablet_replica.cc:333] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532: stopping tablet replica
I20260201 03:03:18.973924 27972 raft_consensus.cc:2243] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:03:18.974155 27972 raft_consensus.cc:2272] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:18.974864 27972 ts_tablet_manager.cc:1916] T 04462ede3f5c45938383ef31924d2f87 P 8c779a097be24011a4b5fb6ddf86a532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:18.974874 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 27506
I20260201 03:03:18.980014 27713 tablet_service.cc:1552] Processing DeleteTablet for tablet 04462ede3f5c45938383ef31924d2f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:18 UTC) from {username='oryx', principal='oryx/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:39382
I20260201 03:03:18.980232 27971 tablet_replica.cc:333] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287: stopping tablet replica
I20260201 03:03:18.980355 27971 raft_consensus.cc:2243] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:18.980465 27971 raft_consensus.cc:2272] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:18.980834 27973 ts_tablet_manager.cc:1916] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:18.981150 27971 ts_tablet_manager.cc:1916] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:18.983325 27973 ts_tablet_manager.cc:1929] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:03:18.983392 27973 log.cc:1199] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/04462ede3f5c45938383ef31924d2f87
I20260201 03:03:18.983367 27971 ts_tablet_manager.cc:1929] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260201 03:03:18.983435 27971 log.cc:1199] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/04462ede3f5c45938383ef31924d2f87
I20260201 03:03:18.983644 27973 ts_tablet_manager.cc:1950] T 04462ede3f5c45938383ef31924d2f87 P 273bae1caac44d809d0ea8a3417710b1: Deleting consensus metadata
I20260201 03:03:18.983711 27971 ts_tablet_manager.cc:1950] T 04462ede3f5c45938383ef31924d2f87 P 02b549b583fb400a89689579b2caa287: Deleting consensus metadata
W20260201 03:03:18.984196 27416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.203.193:43881 (error 108)
I20260201 03:03:18.984279 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 27648
I20260201 03:03:18.985109 27419 catalog_manager.cc:4985] TS 02b549b583fb400a89689579b2caa287 (127.18.203.194:41813): tablet 04462ede3f5c45938383ef31924d2f87 (table test-table [id=c6d41b028a8749a4b4e8c5f4768561a6]) successfully deleted
I20260201 03:03:18.986122 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:18.985618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:43881 (local address 127.0.0.1:40076)
0201 03:03:18.985933 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:18.986014 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.193:43881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":86,"threads_started":1}
I20260201 03:03:18.986290 27416 catalog_manager.cc:4985] TS 273bae1caac44d809d0ea8a3417710b1 (127.18.203.195:33317): tablet 04462ede3f5c45938383ef31924d2f87 (table test-table [id=c6d41b028a8749a4b4e8c5f4768561a6]) successfully deleted
W20260201 03:03:18.986439 27416 catalog_manager.cc:4712] TS 8c779a097be24011a4b5fb6ddf86a532 (127.18.203.193:43881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04462ede3f5c45938383ef31924d2f87: Network error: Client connection negotiation failed: client connection to 127.18.203.193:43881: connect: Connection refused (error 111)
I20260201 03:03:18.992326 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 27790
I20260201 03:03:18.999145 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 27400
2026-02-01T03:03:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4134 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:03:19 dist-test-slave-gf59 krb5kdc[28044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:03:19 dist-test-slave-gf59 krb5kdc[28044](info): set up 2 sockets
Feb 01 03:03:19 dist-test-slave-gf59 krb5kdc[28044](info): commencing operation
krb5kdc: starting...
W20260201 03:03:20.844502 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.822s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:03:20 dist-test-slave-gf59 krb5kdc[28044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:03:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:03:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:20.990041 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:41587
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:41587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:21.101231 28060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:21.101560 28060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:21.101635 28060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:21.106204 28060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:21.106302 28060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:21.106333 28060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:21.106369 28060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:21.106402 28060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:03:21.112634 28060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:41587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:41587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:21.114200 28060 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:21.115355 28060 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:21.122712 28068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:21.122711 28065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:21.122712 28066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:21.123056 28060 server_base.cc:1060] running on GCE node
I20260201 03:03:21.123641 28060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:21.124614 28060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:21.125763 28060 hybrid_clock.cc:648] HybridClock initialized: now 1769915001125739 us; error 47 us; skew 500 ppm
W20260201 03:03:21.128971 28060 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-02-01T03:03:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2139 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:03:21 dist-test-slave-gf59 krb5kdc[28073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:03:21 dist-test-slave-gf59 krb5kdc[28073](info): set up 2 sockets
Feb 01 03:03:21 dist-test-slave-gf59 krb5kdc[28073](info): commencing operation
krb5kdc: starting...
W20260201 03:03:22.976854 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.816s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:03:23 dist-test-slave-gf59 krb5kdc[28073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:03:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:03:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:23.123124 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:35875
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:35875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:23.235733 28089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:23.236033 28089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:23.236119 28089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:23.239470 28089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:23.239568 28089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:23.239629 28089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:23.239662 28089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:23.239692 28089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:03:23.244203 28089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:35875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:35875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:23.245455 28089 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:23.246377 28089 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:23.252276 28095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:23.252300 28094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:23.252336 28097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:23.252676 28089 server_base.cc:1060] running on GCE node
I20260201 03:03:23.253142 28089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:23.254032 28089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:23.255179 28089 hybrid_clock.cc:648] HybridClock initialized: now 1769915003255157 us; error 39 us; skew 500 ppm
Feb 01 03:03:23 dist-test-slave-gf59 krb5kdc[28073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:23.257817 28089 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:03:23.258973 28089 webserver.cc:492] Webserver started at http://127.18.203.254:41341/ using document root <none> and password file <none>
I20260201 03:03:23.259517 28089 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:23.259565 28089 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:23.259789 28089 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:23.261579 28089 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "a460c77191ac460db716e09d679f080f"
format_stamp: "Formatted at 2026-02-01 03:03:23 on dist-test-slave-gf59"
server_key: "f3efc1f37ff49a8ddb3dfe71b3839a40"
server_key_iv: "5e2f30cbb5a539e6e4817aaf66e48b76"
server_key_version: "encryptionkey@0"
I20260201 03:03:23.262042 28089 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "a460c77191ac460db716e09d679f080f"
format_stamp: "Formatted at 2026-02-01 03:03:23 on dist-test-slave-gf59"
server_key: "f3efc1f37ff49a8ddb3dfe71b3839a40"
server_key_iv: "5e2f30cbb5a539e6e4817aaf66e48b76"
server_key_version: "encryptionkey@0"
I20260201 03:03:23.265540 28089 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:03:23.267856 28104 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:23.268949 28089 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260201 03:03:23.269053 28089 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "a460c77191ac460db716e09d679f080f"
format_stamp: "Formatted at 2026-02-01 03:03:23 on dist-test-slave-gf59"
server_key: "f3efc1f37ff49a8ddb3dfe71b3839a40"
server_key_iv: "5e2f30cbb5a539e6e4817aaf66e48b76"
server_key_version: "encryptionkey@0"
I20260201 03:03:23.269145 28089 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:23.294163 28089 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:23.300329 28089 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:23.300519 28089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:23.307740 28089 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:35875
I20260201 03:03:23.307730 28156 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:35875 every 8 connection(s)
I20260201 03:03:23.308774 28089 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:03:23.309288 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 28089
I20260201 03:03:23.309420 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:03:23.309659 19247 external_mini_cluster.cc:1468] Setting key d9c5ebd955deb0a7f117d45b99a9b06a
I20260201 03:03:23.311805 28157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Feb 01 03:03:23 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:03:23.317605 28157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f: Bootstrap starting.
I20260201 03:03:23.319926 28157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:23.320631 28157 log.cc:826] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:23.322271 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:23.310868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.18.203.254:35875)
0201 03:03:23.311235 (+ 367us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:23.311244 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:23.311278 (+ 34us) server_negotiation.cc:408] Connection header received
0201 03:03:23.312390 (+ 1112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:23.312412 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:23.312707 (+ 295us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:23.312983 (+ 276us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:23.313993 (+ 1010us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:23.314746 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:23.315595 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:23.315847 (+ 252us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:23.318183 (+ 2336us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:23.318224 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:23.318234 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:23.318270 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:23.319910 (+ 1640us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:23.320317 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:23.320322 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:23.320326 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:23.320405 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:23.320625 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:23.320628 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:23.320631 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:23.320984 (+ 353us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:23.321184 (+ 200us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:23.321464 (+ 280us) server_negotiation.cc:300] Negotiation successful
0201 03:03:23.321698 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":104,"threads_started":1}
I20260201 03:03:23.322432 28157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f: No bootstrap required, opened a new log
I20260201 03:03:23.325394 28157 raft_consensus.cc:359] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a460c77191ac460db716e09d679f080f" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35875 } }
I20260201 03:03:23.325598 28157 raft_consensus.cc:385] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:23.325661 28157 raft_consensus.cc:740] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a460c77191ac460db716e09d679f080f, State: Initialized, Role: FOLLOWER
I20260201 03:03:23.326089 28157 consensus_queue.cc:260] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a460c77191ac460db716e09d679f080f" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35875 } }
I20260201 03:03:23.326215 28157 raft_consensus.cc:399] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:03:23.326272 28157 raft_consensus.cc:493] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:03:23.326361 28157 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:23.327373 28157 raft_consensus.cc:515] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a460c77191ac460db716e09d679f080f" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35875 } }
I20260201 03:03:23.327731 28157 leader_election.cc:304] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a460c77191ac460db716e09d679f080f; no voters:
I20260201 03:03:23.328007 28157 leader_election.cc:290] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:03:23.328119 28162 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:23.328310 28162 raft_consensus.cc:697] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 1 LEADER]: Becoming Leader. State: Replica: a460c77191ac460db716e09d679f080f, State: Running, Role: LEADER
I20260201 03:03:23.328624 28162 consensus_queue.cc:237] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a460c77191ac460db716e09d679f080f" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35875 } }
I20260201 03:03:23.329032 28157 sys_catalog.cc:565] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:23.330117 28163 sys_catalog.cc:455] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a460c77191ac460db716e09d679f080f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a460c77191ac460db716e09d679f080f" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35875 } } }
I20260201 03:03:23.330247 28163 sys_catalog.cc:458] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:23.330577 28171 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:03:23.330578 28164 sys_catalog.cc:455] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a460c77191ac460db716e09d679f080f. Latest consensus state: current_term: 1 leader_uuid: "a460c77191ac460db716e09d679f080f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a460c77191ac460db716e09d679f080f" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35875 } } }
I20260201 03:03:23.331230 28164 sys_catalog.cc:458] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:23.333696 28171 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:03:23.339095 28171 catalog_manager.cc:1357] Generated new cluster ID: 5b8c8a4fbfb84b79abbadc1fe29e1ac7
I20260201 03:03:23.339172 28171 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:03:23.351761 28171 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:03:23.353014 28171 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:03:23.365038 28171 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f: Generated new TSK 0
I20260201 03:03:23.365927 28171 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:23.415820 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:35875
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:23.524788 28185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:23.525035 28185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:23.525094 28185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:23.528374 28185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:23.528445 28185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:23.528523 28185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:03:23.532874 28185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:35875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:23.533967 28185 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:23.534792 28185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:23.541338 28193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:23.541402 28191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:23.541340 28190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:23.541597 28185 server_base.cc:1060] running on GCE node
I20260201 03:03:23.541949 28185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:23.542500 28185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:23.543689 28185 hybrid_clock.cc:648] HybridClock initialized: now 1769915003543673 us; error 30 us; skew 500 ppm
Feb 01 03:03:23 dist-test-slave-gf59 krb5kdc[28073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:23.546247 28185 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:03:23.547286 28185 webserver.cc:492] Webserver started at http://127.18.203.193:34765/ using document root <none> and password file <none>
I20260201 03:03:23.547840 28185 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:23.547890 28185 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:23.548045 28185 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:23.549716 28185 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "6a4ffdc93637421ebf9d39bbc91cd316"
format_stamp: "Formatted at 2026-02-01 03:03:23 on dist-test-slave-gf59"
server_key: "2ffea71654c827896fd6fb1ad29f65f8"
server_key_iv: "fba36ef7315fc2797f218fbbc536e67d"
server_key_version: "encryptionkey@0"
I20260201 03:03:23.550154 28185 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "6a4ffdc93637421ebf9d39bbc91cd316"
format_stamp: "Formatted at 2026-02-01 03:03:23 on dist-test-slave-gf59"
server_key: "2ffea71654c827896fd6fb1ad29f65f8"
server_key_iv: "fba36ef7315fc2797f218fbbc536e67d"
server_key_version: "encryptionkey@0"
I20260201 03:03:23.553463 28185 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260201 03:03:23.555553 28200 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:23.556560 28185 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:23.556653 28185 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "6a4ffdc93637421ebf9d39bbc91cd316"
format_stamp: "Formatted at 2026-02-01 03:03:23 on dist-test-slave-gf59"
server_key: "2ffea71654c827896fd6fb1ad29f65f8"
server_key_iv: "fba36ef7315fc2797f218fbbc536e67d"
server_key_version: "encryptionkey@0"
I20260201 03:03:23.556746 28185 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:23.572805 28185 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:23.575525 28185 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:23.575732 28185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:23.576309 28185 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:23.577217 28185 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:23.577266 28185 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:23.577311 28185 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:23.577337 28185 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:23.585907 28185 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:40205
I20260201 03:03:23.585940 28313 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:40205 every 8 connection(s)
I20260201 03:03:23.586859 28185 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:03:23.592180 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 28185
I20260201 03:03:23.592294 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:03:23.592519 19247 external_mini_cluster.cc:1468] Setting key 05d48d3c7ee20da345fcd130f8b54fd2
Feb 01 03:03:23 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:03:23.599314 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:23.588589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:38807 (local address 127.18.203.254:35875)
0201 03:03:23.588762 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:23.588767 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:23.589444 (+ 677us) server_negotiation.cc:408] Connection header received
0201 03:03:23.590259 (+ 815us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:23.590262 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:23.590317 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:23.590393 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:23.592091 (+ 1698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:23.592670 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:23.593395 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:23.593580 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:23.596011 (+ 2431us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:23.596031 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:23.596033 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:23.596076 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:23.597438 (+ 1362us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:23.597897 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:23.597900 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:23.597901 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:23.597951 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:23.598211 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:23.598215 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:23.598217 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:23.598408 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:23.598498 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:23.599002 (+ 504us) server_negotiation.cc:300] Negotiation successful
0201 03:03:23.599149 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260201 03:03:23.599970 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:23.588878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:38807)
0201 03:03:23.589289 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:23.589325 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:23.590046 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:23.590542 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:23.590549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:23.590927 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:23.591897 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:23.591932 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:23.592804 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:23.592808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:23.593289 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:23.593298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:23.593476 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:23.594070 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:23.594107 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:23.595850 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:23.597557 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:23.597563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:23.597573 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:23.597803 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:23.598029 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:23.598033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:23.598035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:23.598128 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:23.598510 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:23.598518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:23.598758 (+ 240us) client_negotiation.cc:769] Sending connection context
0201 03:03:23.598947 (+ 189us) client_negotiation.cc:240] Negotiation successful
0201 03:03:23.599221 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":113,"threads_started":1}
I20260201 03:03:23.601320 28314 heartbeater.cc:344] Connected to a master server at 127.18.203.254:35875
I20260201 03:03:23.601564 28314 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:23.602079 28314 heartbeater.cc:507] Master 127.18.203.254:35875 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
I20260201 03:03:23.603547 28121 ts_manager.cc:194] Registered new tserver with Master: 6a4ffdc93637421ebf9d39bbc91cd316 (127.18.203.193:40205)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
I20260201 03:03:23.604825 28121 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:38807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:23.641952 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:35875
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:23.751132 28321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:23.751379 28321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:23.751442 28321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:23.754727 28321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:23.754807 28321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:23.754916 28321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:03:23.759246 28321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:35875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:23.760401 28321 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:23.761324 28321 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:23.767838 28327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:23.767886 28326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:23.767836 28329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:23.768217 28321 server_base.cc:1060] running on GCE node
I20260201 03:03:23.768555 28321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:23.769091 28321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:23.770239 28321 hybrid_clock.cc:648] HybridClock initialized: now 1769915003770224 us; error 35 us; skew 500 ppm
Feb 01 03:03:23 dist-test-slave-gf59 krb5kdc[28073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:23.772868 28321 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:03:23.773989 28321 webserver.cc:492] Webserver started at http://127.18.203.194:38835/ using document root <none> and password file <none>
I20260201 03:03:23.774530 28321 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:23.774583 28321 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:23.774765 28321 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:23.776525 28321 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "06eda0f8ff974d3684be3ba6b4d929e6"
format_stamp: "Formatted at 2026-02-01 03:03:23 on dist-test-slave-gf59"
server_key: "113ce6d7c972e139b6aeb9d30bafbb2f"
server_key_iv: "688d531235fe79063746056ca8cbc7eb"
server_key_version: "encryptionkey@0"
I20260201 03:03:23.776997 28321 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "06eda0f8ff974d3684be3ba6b4d929e6"
format_stamp: "Formatted at 2026-02-01 03:03:23 on dist-test-slave-gf59"
server_key: "113ce6d7c972e139b6aeb9d30bafbb2f"
server_key_iv: "688d531235fe79063746056ca8cbc7eb"
server_key_version: "encryptionkey@0"
I20260201 03:03:23.780527 28321 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:03:23.782637 28336 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:23.783707 28321 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260201 03:03:23.783809 28321 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "06eda0f8ff974d3684be3ba6b4d929e6"
format_stamp: "Formatted at 2026-02-01 03:03:23 on dist-test-slave-gf59"
server_key: "113ce6d7c972e139b6aeb9d30bafbb2f"
server_key_iv: "688d531235fe79063746056ca8cbc7eb"
server_key_version: "encryptionkey@0"
I20260201 03:03:23.783893 28321 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:23.802273 28321 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:23.805502 28321 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:23.805673 28321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:23.806212 28321 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:23.807104 28321 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:23.807154 28321 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:23.807194 28321 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:23.807209 28321 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:23.816183 28321 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:35577
I20260201 03:03:23.816220 28449 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:35577 every 8 connection(s)
I20260201 03:03:23.817186 28321 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:03:23.818161 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 28321
I20260201 03:03:23.818303 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:03:23.818629 19247 external_mini_cluster.cc:1468] Setting key 3b16ccfde358cb139c8493f921859105
Feb 01 03:03:23 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
I20260201 03:03:23.831123 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:23.819216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:51179 (local address 127.18.203.254:35875)
0201 03:03:23.819363 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:23.819368 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:23.820147 (+ 779us) server_negotiation.cc:408] Connection header received
0201 03:03:23.821096 (+ 949us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:23.821100 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:23.821161 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:23.821262 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:23.823247 (+ 1985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:23.823961 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:23.824624 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:23.824824 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:23.827326 (+ 2502us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:23.827352 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:23.827355 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:23.827383 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:23.828728 (+ 1345us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:23.829385 (+ 657us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:23.829389 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:23.829392 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:23.829440 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:23.829802 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:23.829806 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:23.829808 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:23.830037 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:23.830136 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:23.830738 (+ 602us) server_negotiation.cc:300] Negotiation successful
0201 03:03:23.830881 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
I20260201 03:03:23.831913 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:23.819536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:51179)
0201 03:03:23.820009 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:23.820055 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:23.820886 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:23.821422 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:23.821434 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:23.821846 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:23.823072 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:23.823092 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:23.824089 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:23.824092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:23.824505 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:23.824512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:23.824688 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:23.825294 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:23.825321 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:23.827156 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:23.828855 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:23.828862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:23.828875 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:23.829226 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:23.829532 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:23.829536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:23.829538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:23.829690 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:23.830159 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:23.830167 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:23.830457 (+ 290us) client_negotiation.cc:769] Sending connection context
0201 03:03:23.830690 (+ 233us) client_negotiation.cc:240] Negotiation successful
0201 03:03:23.831029 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":110,"threads_started":1}
I20260201 03:03:23.833191 28450 heartbeater.cc:344] Connected to a master server at 127.18.203.254:35875
I20260201 03:03:23.833441 28450 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:23.833979 28450 heartbeater.cc:507] Master 127.18.203.254:35875 requested a full tablet report, sending...
I20260201 03:03:23.835147 28121 ts_manager.cc:194] Registered new tserver with Master: 06eda0f8ff974d3684be3ba6b4d929e6 (127.18.203.194:35577)
I20260201 03:03:23.835777 28121 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:51179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:23.874254 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:35875
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:23.984903 28457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:23.985176 28457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:23.985258 28457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:23.988607 28457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:23.988683 28457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:23.988783 28457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:03:23.993055 28457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:35875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:23.994174 28457 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:23.995050 28457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:24.001802 28462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:24.001789 28465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:24.001945 28457 server_base.cc:1060] running on GCE node
W20260201 03:03:24.001793 28463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:24.002491 28457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:24.003060 28457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:24.004206 28457 hybrid_clock.cc:648] HybridClock initialized: now 1769915004004192 us; error 29 us; skew 500 ppm
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:24.006795 28457 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:03:24.007901 28457 webserver.cc:492] Webserver started at http://127.18.203.195:45729/ using document root <none> and password file <none>
I20260201 03:03:24.008451 28457 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:24.008515 28457 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:24.008735 28457 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:24.010617 28457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "f4b31c482080479b9735d7e92a89c2c3"
format_stamp: "Formatted at 2026-02-01 03:03:24 on dist-test-slave-gf59"
server_key: "cc2e401952d748353f6fa5441ddde695"
server_key_iv: "8613977549945ffa6469f772c804e088"
server_key_version: "encryptionkey@0"
I20260201 03:03:24.011075 28457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "f4b31c482080479b9735d7e92a89c2c3"
format_stamp: "Formatted at 2026-02-01 03:03:24 on dist-test-slave-gf59"
server_key: "cc2e401952d748353f6fa5441ddde695"
server_key_iv: "8613977549945ffa6469f772c804e088"
server_key_version: "encryptionkey@0"
I20260201 03:03:24.014387 28457 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260201 03:03:24.016553 28472 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.017529 28457 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:03:24.017632 28457 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "f4b31c482080479b9735d7e92a89c2c3"
format_stamp: "Formatted at 2026-02-01 03:03:24 on dist-test-slave-gf59"
server_key: "cc2e401952d748353f6fa5441ddde695"
server_key_iv: "8613977549945ffa6469f772c804e088"
server_key_version: "encryptionkey@0"
I20260201 03:03:24.017715 28457 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:24.040103 28457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:24.043049 28457 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:24.043231 28457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:24.043812 28457 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:24.044699 28457 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:24.044747 28457 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.044801 28457 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:24.044828 28457 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.053602 28457 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:35803
I20260201 03:03:24.053633 28585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:35803 every 8 connection(s)
I20260201 03:03:24.054558 28457 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:03:24.061491 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 28457
I20260201 03:03:24.061633 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:03:24.061929 19247 external_mini_cluster.cc:1468] Setting key e6046a3378fd621f15458f6e37f7ccbf
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:03:24.066669 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:24.056288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:37721 (local address 127.18.203.254:35875)
0201 03:03:24.056419 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:24.056423 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:24.057092 (+ 669us) server_negotiation.cc:408] Connection header received
0201 03:03:24.057960 (+ 868us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:24.057964 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:24.058025 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:24.058101 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:24.059946 (+ 1845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.060462 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.061149 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.061331 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.063848 (+ 2517us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:24.063866 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:24.063869 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:24.063900 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:24.065059 (+ 1159us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:24.065500 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:24.065503 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:24.065505 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:24.065552 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:24.065778 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:24.065780 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:24.065781 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:24.065921 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:24.065987 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:24.066440 (+ 453us) server_negotiation.cc:300] Negotiation successful
0201 03:03:24.066549 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260201 03:03:24.067301 28588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:24.056558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:37721)
0201 03:03:24.056964 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:24.057000 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:24.057762 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:24.058273 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:24.058282 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:24.058752 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:24.059803 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.059819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.060578 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.060582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:24.061025 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.061033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.061243 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.061911 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:24.061938 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:24.063710 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:24.065159 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:24.065163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:24.065173 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:24.065409 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:24.065627 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:24.065629 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:24.065631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:24.065716 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:24.065988 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:24.065992 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:24.066241 (+ 249us) client_negotiation.cc:769] Sending connection context
0201 03:03:24.066416 (+ 175us) client_negotiation.cc:240] Negotiation successful
0201 03:03:24.066630 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":108,"threads_started":1}
I20260201 03:03:24.068291 28586 heartbeater.cc:344] Connected to a master server at 127.18.203.254:35875
I20260201 03:03:24.068518 28586 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:24.069005 28586 heartbeater.cc:507] Master 127.18.203.254:35875 requested a full tablet report, sending...
I20260201 03:03:24.069907 28121 ts_manager.cc:194] Registered new tserver with Master: f4b31c482080479b9735d7e92a89c2c3 (127.18.203.195:35803)
I20260201 03:03:24.070446 28121 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:37721
I20260201 03:03:24.075484 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/oryx_127.18.203.193.keytab.
Entry for principal oryx/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/oryx_127.18.203.193.keytab.
WARNING: no policy specified for oryx/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/oryx_127.18.203.194.keytab.
Entry for principal oryx/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/oryx_127.18.203.194.keytab.
WARNING: no policy specified for oryx/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/oryx_127.18.203.195.keytab.
Entry for principal oryx/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/oryx_127.18.203.195.keytab.
I20260201 03:03:24.141572 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 28185
I20260201 03:03:24.147528 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 28321
I20260201 03:03:24.152870 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 28457
I20260201 03:03:24.157879 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 28089
I20260201 03:03:24.164749 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:35875
--webserver_interface=127.18.203.254
--webserver_port=41341
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:35875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:24.274035 28595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:24.274298 28595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:24.274374 28595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:24.278048 28595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:24.278126 28595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:24.278149 28595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:24.278168 28595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:24.278184 28595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:03:24.282593 28595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:35875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:35875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=41341
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:24.283784 28595 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:24.284677 28595 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:24.290311 28600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:24.290416 28601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:24.290580 28595 server_base.cc:1060] running on GCE node
W20260201 03:03:24.290323 28603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:24.291124 28595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:24.292066 28595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:24.293221 28595 hybrid_clock.cc:648] HybridClock initialized: now 1769915004293209 us; error 31 us; skew 500 ppm
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:24.295861 28595 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:03:24.296936 28595 webserver.cc:492] Webserver started at http://127.18.203.254:41341/ using document root <none> and password file <none>
I20260201 03:03:24.297453 28595 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:24.297513 28595 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:24.300875 28595 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:03:24.302765 28610 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.303834 28595 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:03:24.303946 28595 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "a460c77191ac460db716e09d679f080f"
format_stamp: "Formatted at 2026-02-01 03:03:23 on dist-test-slave-gf59"
server_key: "f3efc1f37ff49a8ddb3dfe71b3839a40"
server_key_iv: "5e2f30cbb5a539e6e4817aaf66e48b76"
server_key_version: "encryptionkey@0"
I20260201 03:03:24.304375 28595 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:24.331180 28595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:24.334221 28595 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:24.334405 28595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:24.341892 28662 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:35875 every 8 connection(s)
I20260201 03:03:24.341900 28595 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:35875
I20260201 03:03:24.343061 28595 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:03:24.347527 28663 sys_catalog.cc:263] Verifying existing consensus state
I20260201 03:03:24.349138 28663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f: Bootstrap starting.
I20260201 03:03:24.350668 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 28595
I20260201 03:03:24.351126 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:40205
--local_ip_for_outbound_sockets=127.18.203.193
--tserver_master_addrs=127.18.203.254:35875
--webserver_port=34765
--webserver_interface=127.18.203.193
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.203.193
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/oryx_127.18.203.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260201 03:03:24.365204 28663 log.cc:826] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:24.371466 28663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260201 03:03:24.371848 28663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f: Bootstrap complete.
I20260201 03:03:24.375234 28663 raft_consensus.cc:359] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a460c77191ac460db716e09d679f080f" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35875 } }
I20260201 03:03:24.375578 28663 raft_consensus.cc:740] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a460c77191ac460db716e09d679f080f, State: Initialized, Role: FOLLOWER
I20260201 03:03:24.376082 28663 consensus_queue.cc:260] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a460c77191ac460db716e09d679f080f" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35875 } }
I20260201 03:03:24.376204 28663 raft_consensus.cc:399] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:03:24.376264 28663 raft_consensus.cc:493] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:03:24.376351 28663 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 1 FOLLOWER]: Advancing to term 2
I20260201 03:03:24.377858 28663 raft_consensus.cc:515] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a460c77191ac460db716e09d679f080f" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35875 } }
I20260201 03:03:24.378178 28663 leader_election.cc:304] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a460c77191ac460db716e09d679f080f; no voters:
I20260201 03:03:24.378471 28663 leader_election.cc:290] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260201 03:03:24.378600 28667 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 2 FOLLOWER]: Leader election won for term 2
I20260201 03:03:24.378791 28667 raft_consensus.cc:697] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [term 2 LEADER]: Becoming Leader. State: Replica: a460c77191ac460db716e09d679f080f, State: Running, Role: LEADER
I20260201 03:03:24.379156 28667 consensus_queue.cc:237] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a460c77191ac460db716e09d679f080f" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35875 } }
I20260201 03:03:24.379587 28663 sys_catalog.cc:565] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:24.380424 28669 sys_catalog.cc:455] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a460c77191ac460db716e09d679f080f. Latest consensus state: current_term: 2 leader_uuid: "a460c77191ac460db716e09d679f080f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a460c77191ac460db716e09d679f080f" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35875 } } }
I20260201 03:03:24.380540 28669 sys_catalog.cc:458] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:24.381147 28668 sys_catalog.cc:455] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a460c77191ac460db716e09d679f080f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a460c77191ac460db716e09d679f080f" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35875 } } }
I20260201 03:03:24.381229 28668 sys_catalog.cc:458] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:24.381625 28675 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:03:24.384579 28675 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:03:24.385217 28675 catalog_manager.cc:1269] Loaded cluster ID: 5b8c8a4fbfb84b79abbadc1fe29e1ac7
I20260201 03:03:24.385326 28675 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:03:24.387774 28675 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:03:24.388463 28675 catalog_manager.cc:6038] T 00000000000000000000000000000000 P a460c77191ac460db716e09d679f080f: Loaded TSK: 0
I20260201 03:03:24.389649 28675 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260201 03:03:24.463178 28665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:24.463452 28665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:24.463537 28665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:24.466939 28665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:24.467015 28665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:24.467118 28665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:03:24.471454 28665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/oryx_127.18.203.193.keytab
--principal=oryx/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:40205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=34765
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:35875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:24.472610 28665 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:24.473500 28665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:24.480165 28690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:24.480306 28691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:24.480427 28665 server_base.cc:1060] running on GCE node
W20260201 03:03:24.480165 28693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:24.480954 28665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:24.481614 28665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:24.482777 28665 hybrid_clock.cc:648] HybridClock initialized: now 1769915004482754 us; error 38 us; skew 500 ppm
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915004, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:24.485771 28665 init.cc:377] Logged in from keytab as oryx/127.18.203.193@KRBTEST.COM (short username oryx)
I20260201 03:03:24.486876 28665 webserver.cc:492] Webserver started at http://127.18.203.193:34765/ using document root <none> and password file <none>
I20260201 03:03:24.487437 28665 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:24.487501 28665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:24.491214 28665 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260201 03:03:24.493266 28700 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.494361 28665 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:24.494472 28665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "6a4ffdc93637421ebf9d39bbc91cd316"
format_stamp: "Formatted at 2026-02-01 03:03:23 on dist-test-slave-gf59"
server_key: "2ffea71654c827896fd6fb1ad29f65f8"
server_key_iv: "fba36ef7315fc2797f218fbbc536e67d"
server_key_version: "encryptionkey@0"
I20260201 03:03:24.494890 28665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:24.526670 28665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:24.529698 28665 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:24.529891 28665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:24.530462 28665 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:24.531389 28665 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:24.531441 28665 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.531493 28665 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:24.531517 28665 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.541682 28665 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:40205
I20260201 03:03:24.541704 28813 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:40205 every 8 connection(s)
I20260201 03:03:24.542747 28665 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:03:24.548159 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 28665
I20260201 03:03:24.548601 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:35577
--local_ip_for_outbound_sockets=127.18.203.194
--tserver_master_addrs=127.18.203.254:35875
--webserver_port=38835
--webserver_interface=127.18.203.194
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.203.194
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/oryx_127.18.203.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:24.554157 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:24.544936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:33301)
0201 03:03:24.545394 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:24.545429 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:24.546188 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:24.547232 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:24.547241 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:24.547744 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:24.548734 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.548755 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.549796 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.549800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:24.550210 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.550217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.550451 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.551060 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:24.551089 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:24.553345 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":138,"threads_started":1}
W20260201 03:03:24.554152 28816 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:24.544791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:33301 (local address 127.18.203.254:35875)
0201 03:03:24.545193 (+ 402us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:24.545207 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:24.545539 (+ 332us) server_negotiation.cc:408] Connection header received
0201 03:03:24.546581 (+ 1042us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:24.546591 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:24.546853 (+ 262us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:24.547150 (+ 297us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:24.548910 (+ 1760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.549668 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.550321 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.550574 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.553541 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:33301: BlockingRecv error: recv got EOF from 127.18.203.193:33301 (error 108)
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":99,"threads_started":1}
W20260201 03:03:24.554773 28814 heartbeater.cc:646] Failed to heartbeat to 127.18.203.254:35875 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.203.254:35875: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:24.559566 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:24.555277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:33397)
0201 03:03:24.555392 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:24.555406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:24.555492 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:24.555750 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:24.555753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:24.555896 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:24.556088 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.556094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.556793 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.556796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:24.557175 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.557182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.557282 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.557829 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:24.557841 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:24.559439 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260201 03:03:24.560189 28816 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:24.555346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:33397 (local address 127.18.203.254:35875)
0201 03:03:24.555456 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:24.555460 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:24.555473 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:24.555580 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:24.555583 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:24.555658 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:24.555728 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:24.556199 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.556692 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.557278 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.557461 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.560070 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:33397: BlockingRecv error: recv got EOF from 127.18.203.193:33397 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:24.565058 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:24.560201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:53445)
0201 03:03:24.560312 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:24.560327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:24.560410 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:24.560803 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:24.560805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:24.560937 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:24.561109 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.561115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.562077 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.562080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:24.562509 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.562518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.562616 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.563106 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:24.563120 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:24.564933 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260201 03:03:24.565320 28814 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260201 03:03:24.565466 28816 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:24.560415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:53445 (local address 127.18.203.254:35875)
0201 03:03:24.560589 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:24.560594 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:24.560608 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:03:24.560645 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:24.560648 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:24.560697 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:24.560787 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:24.561221 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.561953 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.562615 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.562775 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.565070 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:53445: BlockingRecv error: recv got EOF from 127.18.203.193:53445 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260201 03:03:24.657573 28818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:24.657810 28818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:24.657869 28818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:24.661168 28818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:24.661264 28818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:24.661365 28818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:03:24.665684 28818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/oryx_127.18.203.194.keytab
--principal=oryx/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:35577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=38835
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:35875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:24.666787 28818 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:24.667671 28818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260201 03:03:24.674612 28818 server_base.cc:1060] running on GCE node
W20260201 03:03:24.674494 28823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:24.674494 28824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:24.674491 28826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:24.675174 28818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:24.675781 28818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:24.676923 28818 hybrid_clock.cc:648] HybridClock initialized: now 1769915004676909 us; error 31 us; skew 500 ppm
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915004, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:24.679674 28818 init.cc:377] Logged in from keytab as oryx/127.18.203.194@KRBTEST.COM (short username oryx)
I20260201 03:03:24.680891 28818 webserver.cc:492] Webserver started at http://127.18.203.194:38835/ using document root <none> and password file <none>
I20260201 03:03:24.681463 28818 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:24.681535 28818 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:24.685355 28818 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:03:24.687356 28833 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.688457 28818 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:24.688565 28818 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "06eda0f8ff974d3684be3ba6b4d929e6"
format_stamp: "Formatted at 2026-02-01 03:03:23 on dist-test-slave-gf59"
server_key: "113ce6d7c972e139b6aeb9d30bafbb2f"
server_key_iv: "688d531235fe79063746056ca8cbc7eb"
server_key_version: "encryptionkey@0"
I20260201 03:03:24.688989 28818 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:24.703970 28818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:24.706624 28818 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:24.706808 28818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:24.707367 28818 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:24.708316 28818 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:24.708369 28818 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.708411 28818 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:24.708439 28818 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.717428 28818 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:35577
I20260201 03:03:24.717450 28946 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:35577 every 8 connection(s)
I20260201 03:03:24.718482 28818 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:03:24.725261 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 28818
I20260201 03:03:24.725813 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:35803
--local_ip_for_outbound_sockets=127.18.203.195
--tserver_master_addrs=127.18.203.254:35875
--webserver_port=45729
--webserver_interface=127.18.203.195
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.203.195
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/oryx_127.18.203.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
W20260201 03:03:24.729514 28816 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:24.720235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:41431 (local address 127.18.203.254:35875)
0201 03:03:24.720386 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:24.720391 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:24.721109 (+ 718us) server_negotiation.cc:408] Connection header received
0201 03:03:24.721920 (+ 811us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:24.721923 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:24.721978 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:24.722066 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:24.723745 (+ 1679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.724532 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.725459 (+ 927us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.725774 (+ 315us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.729374 (+ 3600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:41431: BlockingRecv error: recv got EOF from 127.18.203.194:41431 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260201 03:03:24.730010 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:24.720516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:41431)
0201 03:03:24.720969 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:24.721006 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:24.721722 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:24.722221 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:24.722229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:24.722605 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:24.723543 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.723557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.724664 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.724669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:24.725272 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.725282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.725550 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.726224 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:24.726262 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:24.729188 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
W20260201 03:03:24.730877 28947 heartbeater.cc:646] Failed to heartbeat to 127.18.203.254:35875 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.203.254:35875: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:24.736644 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:24.731649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:33811)
0201 03:03:24.732130 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:24.732144 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:24.732240 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:24.732635 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:24.732638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:24.732773 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:24.732956 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.732962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.733682 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.733685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:24.734076 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.734084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.734194 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.734716 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:24.734728 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:24.736518 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386}
W20260201 03:03:24.736847 28816 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:24.731583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:33811 (local address 127.18.203.254:35875)
0201 03:03:24.731875 (+ 292us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:24.731880 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:24.732323 (+ 443us) server_negotiation.cc:408] Connection header received
0201 03:03:24.732371 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:24.732374 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:24.732435 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:24.732508 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:24.733077 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.733578 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.734187 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.734370 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.736661 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:33811: BlockingRecv error: recv got EOF from 127.18.203.194:33811 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:24.741519 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:24.737306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:44531)
0201 03:03:24.737435 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:24.737448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:24.737511 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:24.737720 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:24.737722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:24.737837 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:24.737983 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.737988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.738916 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.738918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:24.739253 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.739258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.739333 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.739774 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:24.739786 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:24.741426 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260201 03:03:24.741695 28816 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:24.737387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:44531 (local address 127.18.203.254:35875)
0201 03:03:24.737513 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:24.737517 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:24.737531 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:03:24.737571 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:24.737573 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:24.737623 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:24.737713 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:24.738087 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.738820 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.739348 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.739532 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.741584 (+ 2052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:44531: BlockingRecv error: recv got EOF from 127.18.203.194:44531 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260201 03:03:24.741904 28947 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260201 03:03:24.839731 28950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:24.839984 28950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:24.840054 28950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:24.843302 28950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:24.843379 28950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:24.843459 28950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:03:24.847684 28950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/krb5kdc/oryx_127.18.203.195.keytab
--principal=oryx/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:35803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=45729
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:35875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:24.848775 28950 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:24.849630 28950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:24.856143 28956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:24.856141 28958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:24.856370 28950 server_base.cc:1060] running on GCE node
W20260201 03:03:24.856163 28955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:24.856781 28950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:24.857379 28950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:24.858532 28950 hybrid_clock.cc:648] HybridClock initialized: now 1769915004858509 us; error 39 us; skew 500 ppm
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915004, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:24.861183 28950 init.cc:377] Logged in from keytab as oryx/127.18.203.195@KRBTEST.COM (short username oryx)
I20260201 03:03:24.862284 28950 webserver.cc:492] Webserver started at http://127.18.203.195:45729/ using document root <none> and password file <none>
I20260201 03:03:24.862804 28950 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:24.862861 28950 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:24.866356 28950 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:24.868213 28965 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.869232 28950 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:24.869325 28950 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "f4b31c482080479b9735d7e92a89c2c3"
format_stamp: "Formatted at 2026-02-01 03:03:24 on dist-test-slave-gf59"
server_key: "cc2e401952d748353f6fa5441ddde695"
server_key_iv: "8613977549945ffa6469f772c804e088"
server_key_version: "encryptionkey@0"
I20260201 03:03:24.869730 28950 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:24.883723 28950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:24.886157 28950 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:24.886320 28950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:24.886818 28950 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:24.887730 28950 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:24.887780 28950 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.887826 28950 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:24.887845 28950 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:24.896694 28950 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:35803
I20260201 03:03:24.896719 29078 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:35803 every 8 connection(s)
I20260201 03:03:24.897723 28950 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:03:24.904098 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 28950
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
W20260201 03:03:24.907717 28816 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:24.899345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:48035 (local address 127.18.203.254:35875)
0201 03:03:24.899476 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:24.899481 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:24.900194 (+ 713us) server_negotiation.cc:408] Connection header received
0201 03:03:24.901057 (+ 863us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:24.901060 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:24.901123 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:24.901201 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:24.902854 (+ 1653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.903371 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.904132 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.904313 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.907580 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:48035: BlockingRecv error: recv got EOF from 127.18.203.195:48035 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260201 03:03:24.908581 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:24.899601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:48035)
0201 03:03:24.900051 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:24.900088 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:24.900850 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:24.901379 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:24.901387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:24.901769 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:24.902707 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.902721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.903488 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.903491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:24.904003 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.904014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.904252 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.904897 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:24.904933 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:24.907471 (+ 2538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":105,"threads_started":1}
W20260201 03:03:24.909025 29079 heartbeater.cc:646] Failed to heartbeat to 127.18.203.254:35875 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.203.254:35875: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:24.913895 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:24.905088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34156 (local address 127.18.203.254:35875)
0201 03:03:24.905369 (+ 281us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:24.905373 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:24.905387 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:03:24.905423 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:24.905426 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:24.905474 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:24.905574 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:24.906490 (+ 916us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.907023 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.907947 (+ 924us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.908103 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.909242 (+ 1139us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:24.909261 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:24.909272 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:24.909312 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:24.911144 (+ 1832us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:24.911519 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:24.911526 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:24.911531 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:24.911674 (+ 143us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:24.912730 (+ 1056us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:24.912734 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:24.912735 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:24.913065 (+ 330us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:24.913270 (+ 205us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:24.913517 (+ 247us) server_negotiation.cc:300] Negotiation successful
0201 03:03:24.913744 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":97,"threads_started":1}
I20260201 03:03:24.914202 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:24.909455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:37233)
0201 03:03:24.909801 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:24.909813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:24.909889 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:24.910126 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:24.910130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:24.910270 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:24.910523 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.910558 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.911253 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.911257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:24.911717 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.911725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.911821 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.912267 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:24.912293 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:24.914087 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260201 03:03:24.914412 28816 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:24.909557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:37233 (local address 127.18.203.254:35875)
0201 03:03:24.909682 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:24.909686 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:24.909865 (+ 179us) server_negotiation.cc:408] Connection header received
0201 03:03:24.909969 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:24.909972 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:24.910016 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:24.910088 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:24.910670 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.911147 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.911842 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.912074 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.914315 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:37233: BlockingRecv error: recv got EOF from 127.18.203.195:37233 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:24 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:24.919013 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:24.914662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:37107)
0201 03:03:24.914768 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:24.914781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:24.914850 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:24.915760 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:24.915763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:24.915882 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:24.916026 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.916031 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.916667 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.916670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:24.917009 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:24.917015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.917106 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.917483 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:24.917494 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:24.918903 (+ 1409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260201 03:03:24.919104 28816 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:24.915412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:37107 (local address 127.18.203.254:35875)
0201 03:03:24.915540 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:24.915544 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:24.915555 (+ 11us) server_negotiation.cc:408] Connection header received
0201 03:03:24.915584 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:24.915588 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:24.915656 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:24.915718 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:24.916128 (+ 410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.916575 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:24.917104 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:24.917264 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:24.919018 (+ 1754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:37107: BlockingRecv error: recv got EOF from 127.18.203.195:37107 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260201 03:03:24.919209 29079 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Feb 01 03:03:25 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:25 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:25.570928 29084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:25.565964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:57075)
0201 03:03:25.566256 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:25.566270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:25.566381 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:25.566645 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:25.566648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:25.566871 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:25.567155 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:25.567164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:25.567996 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:25.567999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:25.568418 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:25.568425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:25.568527 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:25.569107 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:25.569130 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:25.570800 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":93,"threads_started":1}
W20260201 03:03:25.571050 29085 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:25.566047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:57075 (local address 127.18.203.254:35875)
0201 03:03:25.566344 (+ 297us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:25.566347 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:25.566360 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:25.566475 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:25.566478 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:25.566538 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:25.566612 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:25.567264 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:25.567879 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:25.568526 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:25.568703 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:25.570926 (+ 2223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:57075: BlockingRecv error: recv got EOF from 127.18.203.193:57075 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
Feb 01 03:03:25 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:25 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:25.747104 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:25.742509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:44473)
0201 03:03:25.742760 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:25.742773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:25.742872 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:25.743122 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:25.743124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:25.743316 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:25.743517 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:25.743522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:25.744213 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:25.744216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:25.744607 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:25.744613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:25.744691 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:25.745221 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:25.745235 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:25.746953 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
W20260201 03:03:25.747195 29085 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:25.742541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:44473 (local address 127.18.203.254:35875)
0201 03:03:25.742670 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:25.742674 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:25.742829 (+ 155us) server_negotiation.cc:408] Connection header received
0201 03:03:25.742954 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:25.742956 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:25.743002 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:25.743064 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:25.743628 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:25.744114 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:25.744688 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:25.744811 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:25.747087 (+ 2276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:44473: BlockingRecv error: recv got EOF from 127.18.203.194:44473 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Feb 01 03:03:25 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:25 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:25.924504 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:25.919848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:60771)
0201 03:03:25.920136 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:25.920149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:25.920263 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:25.920480 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:25.920482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:25.920677 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:25.920875 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:25.920880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:25.921581 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:25.921584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:25.921978 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:25.921985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:25.922079 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:25.922628 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:25.922641 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:25.924374 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
W20260201 03:03:25.924604 29085 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:25.919919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:60771 (local address 127.18.203.254:35875)
0201 03:03:25.920054 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:25.920058 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:25.920214 (+ 156us) server_negotiation.cc:408] Connection header received
0201 03:03:25.920335 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:25.920338 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:25.920388 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:25.920456 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:25.920983 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:25.921482 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:25.922072 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:25.922205 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:25.924507 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:60771: BlockingRecv error: recv got EOF from 127.18.203.195:60771 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Feb 01 03:03:26 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:26 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:26.576728 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:26.571810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:36293)
0201 03:03:26.572141 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:26.572156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:26.572266 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:26.572549 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:26.572552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:26.572778 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:26.573044 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:26.573054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:26.573778 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:26.573781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:26.574243 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:26.574252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:26.574349 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:26.574862 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:26.574875 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:26.576606 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
W20260201 03:03:26.576853 29089 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:26.571933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:36293 (local address 127.18.203.254:35875)
0201 03:03:26.572212 (+ 279us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:26.572216 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:26.572231 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:03:26.572374 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:26.572377 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:26.572426 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:26.572506 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:26.573164 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:26.573677 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:26.574347 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:26.574486 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:26.576742 (+ 2256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:36293: BlockingRecv error: recv got EOF from 127.18.203.193:36293 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
Feb 01 03:03:26 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:26 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:26.753109 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:26.747987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:41119)
0201 03:03:26.748300 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:26.748325 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:26.748432 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:26.748712 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:26.748714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:26.748920 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:26.749156 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:26.749165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:26.750121 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:26.750125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:26.750610 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:26.750620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:26.750712 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:26.751231 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:26.751244 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:26.752986 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":140,"threads_started":1}
W20260201 03:03:26.753285 29089 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:26.748072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:41119 (local address 127.18.203.254:35875)
0201 03:03:26.748219 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:26.748223 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:26.748389 (+ 166us) server_negotiation.cc:408] Connection header received
0201 03:03:26.748543 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:26.748547 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:26.748603 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:26.748681 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:26.749283 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:26.749957 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:26.750722 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:26.750878 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:26.753163 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:41119: BlockingRecv error: recv got EOF from 127.18.203.194:41119 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Feb 01 03:03:26 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:26 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:26.930155 29091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:26.925357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:46301)
0201 03:03:26.925634 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:26.925650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:26.925771 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:26.926049 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:26.926052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:26.926269 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:26.926483 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:26.926491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:26.927184 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:26.927187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:26.927579 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:26.927587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:26.927700 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:26.928223 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:26.928238 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:26.930006 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":87,"threads_started":1}
W20260201 03:03:26.930299 29089 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:26.925467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:46301 (local address 127.18.203.254:35875)
0201 03:03:26.925617 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:26.925622 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:26.925716 (+ 94us) server_negotiation.cc:408] Connection header received
0201 03:03:26.925853 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:26.925857 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:26.925917 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:26.925990 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:26.926587 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:26.927078 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:26.927689 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:26.927818 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:26.930163 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:46301: BlockingRecv error: recv got EOF from 127.18.203.195:46301 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Feb 01 03:03:27 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:27 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:27.582566 29093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:27.577525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:49167)
0201 03:03:27.577846 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:27.577862 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:27.577985 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:27.578267 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:27.578271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:27.578458 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:27.578660 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:27.578666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:27.579355 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:27.579358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:27.579776 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:27.579783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:27.579869 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:27.580380 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:27.580392 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:27.582442 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":116,"threads_started":1}
W20260201 03:03:27.582751 29092 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:27.577574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:49167 (local address 127.18.203.254:35875)
0201 03:03:27.577861 (+ 287us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:27.577865 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:27.577922 (+ 57us) server_negotiation.cc:408] Connection header received
0201 03:03:27.578080 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:27.578084 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:27.578142 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:27.578243 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:27.578757 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:27.579263 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:27.579868 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:27.580006 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:27.582610 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:49167: BlockingRecv error: recv got EOF from 127.18.203.193:49167 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
Feb 01 03:03:27 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:27 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:27.758505 29094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:27.753854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:35869)
0201 03:03:27.754166 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:27.754180 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:27.754285 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:27.754512 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:27.754514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:27.754699 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:27.754895 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:27.754900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:27.755591 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:27.755594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:27.756026 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:27.756042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:27.756129 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:27.756630 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:27.756643 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:27.758375 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":78,"threads_started":1}
W20260201 03:03:27.758599 29092 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:27.753934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:35869 (local address 127.18.203.254:35875)
0201 03:03:27.754084 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:27.754088 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:27.754244 (+ 156us) server_negotiation.cc:408] Connection header received
0201 03:03:27.754358 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:27.754361 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:27.754419 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:27.754486 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:27.754982 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:27.755487 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:27.756124 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:27.756251 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:27.758503 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:35869: BlockingRecv error: recv got EOF from 127.18.203.194:35869 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Feb 01 03:03:27 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:27 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:27.935707 29095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:27.930847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:34037)
0201 03:03:27.931121 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:27.931148 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:27.931256 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:27.931496 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:27.931499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:27.931745 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:27.931939 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:27.931946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:27.932673 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:27.932677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:27.933052 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:27.933059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:27.933171 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:27.933711 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:27.933724 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:27.935516 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":95,"threads_started":1}
W20260201 03:03:27.935810 29092 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:27.930954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:34037 (local address 127.18.203.254:35875)
0201 03:03:27.931095 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:27.931099 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:27.931210 (+ 111us) server_negotiation.cc:408] Connection header received
0201 03:03:27.931338 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:27.931341 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:27.931397 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:27.931464 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:27.932066 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:27.932563 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:27.933165 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:27.933339 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:27.935706 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:34037: BlockingRecv error: recv got EOF from 127.18.203.195:34037 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Feb 01 03:03:28 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:28 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:28.590977 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:28.583326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:45005)
0201 03:03:28.583721 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:28.583735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:28.583840 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:28.584086 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:28.584088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:28.584268 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:28.584479 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:28.584485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:28.585202 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:28.585205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:28.585596 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:28.585602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:28.585691 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:28.588972 (+ 3281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:28.588987 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:28.590817 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":109,"threads_started":1}
W20260201 03:03:28.591078 29097 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:28.583406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:45005 (local address 127.18.203.254:35875)
0201 03:03:28.583783 (+ 377us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:28.583787 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:28.583802 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:03:28.583937 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:28.583941 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:28.583989 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:28.584065 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:28.584582 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:28.585106 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:28.585675 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:28.585803 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:28.590962 (+ 5159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:45005: BlockingRecv error: recv got EOF from 127.18.203.193:45005 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":60,"threads_started":1}
Feb 01 03:03:28 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:28 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:28.764060 29098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:28.759297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:47799)
0201 03:03:28.759599 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:28.759635 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:28.759735 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:28.759954 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:28.759956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:28.760161 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:28.760359 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:28.760365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:28.761022 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:28.761024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:28.761411 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:28.761417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:28.761500 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:28.762048 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:28.762064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:28.763908 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
W20260201 03:03:28.764129 29097 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:28.759343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:47799 (local address 127.18.203.254:35875)
0201 03:03:28.759467 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:28.759471 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:28.759693 (+ 222us) server_negotiation.cc:408] Connection header received
0201 03:03:28.759817 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:28.759820 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:28.759867 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:28.759924 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:28.760455 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:28.760947 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:28.761491 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:28.761623 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:28.764037 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:47799: BlockingRecv error: recv got EOF from 127.18.203.194:47799 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Feb 01 03:03:28 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:28 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:28.941159 29099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:28.936492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:54381)
0201 03:03:28.936772 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:28.936784 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:28.936891 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:28.937122 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:28.937125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:28.937332 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:28.937536 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:28.937543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:28.938245 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:28.938249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:28.938634 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:28.938641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:28.938740 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:28.939277 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:28.939290 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:28.941013 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":110,"threads_started":1}
W20260201 03:03:28.941259 29097 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:28.936561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:54381 (local address 127.18.203.254:35875)
0201 03:03:28.936691 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:28.936695 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:28.936846 (+ 151us) server_negotiation.cc:408] Connection header received
0201 03:03:28.936971 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:28.936975 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:28.937023 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:28.937090 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:28.937650 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:28.938142 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:28.938734 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:28.938904 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:28.941160 (+ 2256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:54381: BlockingRecv error: recv got EOF from 127.18.203.195:54381 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Feb 01 03:03:29 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:29 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:29.597234 29100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:29.591847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:51989)
0201 03:03:29.592188 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:29.592205 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:29.592359 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:29.592588 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:29.592590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:29.592793 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:29.593015 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:29.593020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:29.593786 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:29.593789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:29.594166 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:29.594173 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:29.594280 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:29.594964 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:29.594978 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:29.597106 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
W20260201 03:03:29.597362 29101 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:29.591961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:51989 (local address 127.18.203.254:35875)
0201 03:03:29.592219 (+ 258us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:29.592222 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:29.592282 (+ 60us) server_negotiation.cc:408] Connection header received
0201 03:03:29.592440 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:29.592442 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:29.592486 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:29.592564 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:29.593137 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:29.593648 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:29.594275 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:29.594455 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:29.597232 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:51989: BlockingRecv error: recv got EOF from 127.18.203.193:51989 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":60,"threads_started":1}
Feb 01 03:03:29 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:29 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:29.770177 29102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:29.764858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:47157)
0201 03:03:29.765219 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:29.765237 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:29.765370 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:29.765652 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:29.765655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:29.765914 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:29.766123 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:29.766131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:29.766904 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:29.766908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:29.767348 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:29.767356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:29.767472 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:29.768053 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:29.768066 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:29.770021 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":86,"threads_started":1}
W20260201 03:03:29.770252 29101 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:29.765012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:47157 (local address 127.18.203.254:35875)
0201 03:03:29.765224 (+ 212us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:29.765229 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:29.765321 (+ 92us) server_negotiation.cc:408] Connection header received
0201 03:03:29.765483 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:29.765487 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:29.765542 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:29.765616 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:29.766264 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:29.766786 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:29.767469 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:29.767707 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:29.770155 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:47157: BlockingRecv error: recv got EOF from 127.18.203.194:47157 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Feb 01 03:03:29 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:29 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:29.947227 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:29.942039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:37933)
0201 03:03:29.942378 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:29.942392 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:29.942501 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:29.942743 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:29.942745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:29.942983 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:29.943207 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:29.943214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:29.944054 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:29.944057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:29.944469 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:29.944477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:29.944579 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:29.945133 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:29.945147 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:29.947099 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
W20260201 03:03:29.947336 29101 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:29.942171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:37933 (local address 127.18.203.254:35875)
0201 03:03:29.942357 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:29.942361 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:29.942495 (+ 134us) server_negotiation.cc:408] Connection header received
0201 03:03:29.942593 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:29.942596 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:29.942653 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:29.942717 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:29.943357 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:29.943949 (+ 592us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:29.944599 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:29.944786 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:29.947236 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:37933: BlockingRecv error: recv got EOF from 127.18.203.195:37933 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Feb 01 03:03:30 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:30 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:30.603039 29104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:30.597984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:52611)
0201 03:03:30.598425 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:30.598441 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:30.598567 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:30.598816 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:30.598819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:30.599040 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:30.599246 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:30.599253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:30.600041 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:30.600044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:30.600439 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:30.600445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:30.600547 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:30.601067 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:30.601084 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:30.602901 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":94,"threads_started":1}
W20260201 03:03:30.603163 29105 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:30.598096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:52611 (local address 127.18.203.254:35875)
0201 03:03:30.598354 (+ 258us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:30.598359 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:30.598507 (+ 148us) server_negotiation.cc:408] Connection header received
0201 03:03:30.598655 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:30.598658 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:30.598707 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:30.598783 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:30.599365 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:30.599925 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:30.600547 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:30.600725 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:30.603053 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:52611: BlockingRecv error: recv got EOF from 127.18.203.193:52611 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
Feb 01 03:03:30 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:30 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:30.775744 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:30.770942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:46821)
0201 03:03:30.771191 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:30.771203 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:30.771322 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:30.771705 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:30.771707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:30.771910 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:30.772112 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:30.772118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:30.772877 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:30.772879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:30.773258 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:30.773264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:30.773343 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:30.773834 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:30.773847 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:30.775564 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
W20260201 03:03:30.775825 29105 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:30.770975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:46821 (local address 127.18.203.254:35875)
0201 03:03:30.771115 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:30.771120 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:30.771269 (+ 149us) server_negotiation.cc:408] Connection header received
0201 03:03:30.771410 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:30.771413 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:30.771574 (+ 161us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:30.771681 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:30.772269 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:30.772793 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:30.773333 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:30.773467 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:30.775729 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:46821: BlockingRecv error: recv got EOF from 127.18.203.194:46821 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Feb 01 03:03:30 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:30 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:30.952667 29107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:30.948082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:60411)
0201 03:03:30.948368 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:30.948381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:30.948491 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:30.948711 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:30.948713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:30.948894 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:30.949085 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:30.949090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:30.949735 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:30.949738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:30.950104 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:30.950109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:30.950187 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:30.950699 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:30.950713 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:30.952515 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":118,"threads_started":1}
W20260201 03:03:30.952762 29105 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:30.948151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:60411 (local address 127.18.203.254:35875)
0201 03:03:30.948277 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:30.948281 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:30.948481 (+ 200us) server_negotiation.cc:408] Connection header received
0201 03:03:30.948574 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:30.948576 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:30.948627 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:30.948689 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:30.949176 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:30.949659 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:30.950204 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:30.950340 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:30.952656 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:60411: BlockingRecv error: recv got EOF from 127.18.203.195:60411 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Feb 01 03:03:31 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:31 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:31.609130 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:31.603888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:48131)
0201 03:03:31.604238 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:31.604252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:31.604350 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:31.604601 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:31.604604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:31.604789 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:31.604985 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:31.604991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:31.605779 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:31.605781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:31.606168 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:31.606173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:31.606258 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:31.606794 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:31.606807 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:31.608968 (+ 2161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":78,"threads_started":1}
W20260201 03:03:31.609272 29109 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:31.603975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:48131 (local address 127.18.203.254:35875)
0201 03:03:31.604274 (+ 299us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:31.604277 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:31.604296 (+ 19us) server_negotiation.cc:408] Connection header received
0201 03:03:31.604443 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:31.604445 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:31.604496 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:31.604580 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:31.605140 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:31.605692 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:31.606268 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:31.606426 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:31.609146 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:48131: BlockingRecv error: recv got EOF from 127.18.203.193:48131 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":116,"threads_started":1}
Feb 01 03:03:31 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:31 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:31.781836 29110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:31.776570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:34829)
0201 03:03:31.776906 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:31.776922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:31.777040 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:31.777300 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:31.777303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:31.777517 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:31.777707 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:31.777714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:31.778437 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:31.778441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:31.778811 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:31.778818 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:31.778916 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:31.779480 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:31.779492 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:31.781682 (+ 2190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":88,"threads_started":1}
W20260201 03:03:31.781916 29109 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:31.776698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:34829 (local address 127.18.203.254:35875)
0201 03:03:31.776882 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:31.776887 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:31.776993 (+ 106us) server_negotiation.cc:408] Connection header received
0201 03:03:31.777143 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:31.777146 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:31.777198 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:31.777265 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:31.777825 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:31.778335 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:31.778917 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:31.779111 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:31.781815 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:34829: BlockingRecv error: recv got EOF from 127.18.203.194:34829 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Feb 01 03:03:31 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:31 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:31.958084 29111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:31.953474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:48437)
0201 03:03:31.953767 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:31.953780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:31.953883 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:31.954123 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:31.954125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:31.954308 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:31.954511 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:31.954517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:31.955240 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:31.955243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:31.955660 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:31.955666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:31.955748 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:31.956248 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:31.956261 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:31.957962 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":74,"threads_started":1}
W20260201 03:03:31.958276 29109 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:31.953588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:48437 (local address 127.18.203.254:35875)
0201 03:03:31.953724 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:31.953728 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:31.953842 (+ 114us) server_negotiation.cc:408] Connection header received
0201 03:03:31.953953 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:31.953956 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:31.954003 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:31.954108 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:31.954625 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:31.955155 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:31.955760 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:31.955906 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:31.958152 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:48437: BlockingRecv error: recv got EOF from 127.18.203.195:48437 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Feb 01 03:03:32 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:32 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:32.615031 29112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:32.609880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:34669)
0201 03:03:32.610156 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:32.610171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:32.610267 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:32.610533 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:32.610536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:32.610735 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:32.610937 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:32.610943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:32.611773 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:32.611777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:32.612391 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:32.612402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:32.612528 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:32.613061 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:32.613075 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:32.614880 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
W20260201 03:03:32.615216 29113 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:32.610029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:34669 (local address 127.18.203.254:35875)
0201 03:03:32.610272 (+ 243us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:32.610276 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:32.610290 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:03:32.610330 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:32.610332 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:32.610378 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:32.610460 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:32.611081 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:32.611669 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:32.612502 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:32.612647 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:32.615066 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:34669: BlockingRecv error: recv got EOF from 127.18.203.193:34669 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
Feb 01 03:03:32 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:32 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:32.788146 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:32.782811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:60855)
0201 03:03:32.783160 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:32.783175 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:32.783305 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:32.783588 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:32.783592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:32.783852 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:32.784080 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:32.784088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:32.785164 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:32.785168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:32.785556 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:32.785563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:32.785658 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:32.786176 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:32.786189 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:32.788015 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":109,"threads_started":1}
W20260201 03:03:32.788262 29113 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:32.782912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:60855 (local address 127.18.203.254:35875)
0201 03:03:32.783146 (+ 234us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:32.783151 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:32.783257 (+ 106us) server_negotiation.cc:408] Connection header received
0201 03:03:32.783397 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:32.783401 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:32.783454 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:32.783534 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:32.784206 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:32.785043 (+ 837us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:32.785645 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:32.785776 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:32.788161 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:60855: BlockingRecv error: recv got EOF from 127.18.203.194:60855 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Feb 01 03:03:32 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:32 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:32.964287 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:32.958963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:36401)
0201 03:03:32.959266 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:32.959279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:32.959406 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:32.959758 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:32.959760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:32.959939 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:32.960178 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:32.960188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:32.961044 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:32.961047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:32.961503 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:32.961513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:32.961613 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:32.962128 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:32.962142 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:32.964164 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":114,"threads_started":1}
W20260201 03:03:32.964497 29113 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:32.959121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:36401 (local address 127.18.203.254:35875)
0201 03:03:32.959263 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:32.959266 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:32.959351 (+ 85us) server_negotiation.cc:408] Connection header received
0201 03:03:32.959538 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:32.959542 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:32.959596 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:32.959737 (+ 141us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:32.960300 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:32.960919 (+ 619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:32.961637 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:32.961780 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:32.964353 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:36401: BlockingRecv error: recv got EOF from 127.18.203.195:36401 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Feb 01 03:03:33 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:33 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:33.624070 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:33.616064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:46185)
0201 03:03:33.616406 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:33.616422 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:33.616565 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:33.616863 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:33.616866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:33.617134 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:33.617390 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:33.617398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:33.618130 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:33.618133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:33.618509 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:33.618515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:33.618596 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:33.622022 (+ 3426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:33.622036 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:33.623941 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":88,"threads_started":1}
W20260201 03:03:33.624204 29117 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:33.616179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:46185 (local address 127.18.203.254:35875)
0201 03:03:33.616442 (+ 263us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:33.616447 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:33.616498 (+ 51us) server_negotiation.cc:408] Connection header received
0201 03:03:33.616666 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:33.616669 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:33.616730 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:33.616816 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:33.617508 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:33.618025 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:33.618586 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:33.618711 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:33.624089 (+ 5378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:46185: BlockingRecv error: recv got EOF from 127.18.203.193:46185 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":60,"threads_started":1}
Feb 01 03:03:33 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:33 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:33.794160 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:33.788974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:53693)
0201 03:03:33.789335 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:33.789349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:33.789452 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:33.789688 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:33.789690 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:33.789881 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:33.790077 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:33.790084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:33.790859 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:33.790863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:33.791280 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:33.791288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:33.791408 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:33.792012 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:33.792028 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:33.794033 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":114,"threads_started":1}
W20260201 03:03:33.794296 29117 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:33.789089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:53693 (local address 127.18.203.254:35875)
0201 03:03:33.789246 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:33.789250 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:33.789413 (+ 163us) server_negotiation.cc:408] Connection header received
0201 03:03:33.789528 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:33.789531 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:33.789585 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:33.789664 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:33.790220 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:33.790734 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:33.791417 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:33.791600 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:33.794185 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:53693: BlockingRecv error: recv got EOF from 127.18.203.194:53693 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Feb 01 03:03:33 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:33 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:33.970254 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:33.965111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:47205)
0201 03:03:33.965461 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:33.965477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:33.965612 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:33.965887 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:33.965890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:33.966103 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:33.966298 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:33.966305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:33.967116 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:33.967120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:33.967565 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:33.967575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:33.967723 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:33.968311 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:33.968324 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:33.970125 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":123,"threads_started":1}
W20260201 03:03:33.970396 29117 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:33.965263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:47205 (local address 127.18.203.254:35875)
0201 03:03:33.965461 (+ 198us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:33.965466 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:33.965558 (+ 92us) server_negotiation.cc:408] Connection header received
0201 03:03:33.965709 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:33.965713 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:33.965763 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:33.965834 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:33.966449 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:33.967005 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:33.967707 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:33.967906 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:33.970274 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:47205: BlockingRecv error: recv got EOF from 127.18.203.195:47205 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Feb 01 03:03:34 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:34 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:34.629964 29120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:34.624968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:36999)
0201 03:03:34.625239 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:34.625253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:34.625347 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:34.625600 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:34.625603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:34.625799 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:34.625995 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:34.626000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:34.626815 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:34.626820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:34.627301 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:34.627314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:34.627440 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:34.627990 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:34.628006 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:34.629813 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
W20260201 03:03:34.630060 29121 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:34.625040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:36999 (local address 127.18.203.254:35875)
0201 03:03:34.625321 (+ 281us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:34.625325 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:34.625338 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:34.625433 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:34.625435 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:34.625481 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:34.625558 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:34.626117 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:34.626689 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:34.627421 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:34.627561 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:34.629953 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:36999: BlockingRecv error: recv got EOF from 127.18.203.193:36999 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":116,"threads_started":1}
Feb 01 03:03:34 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:34 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:34.800091 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:34.794912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:35383)
0201 03:03:34.795279 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:34.795293 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:34.795423 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:34.795723 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:34.795726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:34.795949 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:34.796161 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:34.796169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:34.796969 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:34.796973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:34.797380 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:34.797388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:34.797508 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:34.798056 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:34.798070 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:34.799939 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":111,"threads_started":1}
W20260201 03:03:34.800200 29121 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:34.794998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:35383 (local address 127.18.203.254:35875)
0201 03:03:34.795160 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:34.795166 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:34.795386 (+ 220us) server_negotiation.cc:408] Connection header received
0201 03:03:34.795535 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:34.795538 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:34.795589 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:34.795688 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:34.796311 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:34.796836 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:34.797500 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:34.797693 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:34.800100 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:35383: BlockingRecv error: recv got EOF from 127.18.203.194:35383 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Feb 01 03:03:34 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:34 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:34.976157 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:34.970978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:57939)
0201 03:03:34.971309 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:34.971327 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:34.971455 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:34.971729 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:34.971732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:34.971939 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:34.972144 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:34.972151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:34.972877 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:34.972880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:34.973265 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:34.973272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:34.973375 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:34.973939 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:34.973953 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:34.976009 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":110,"threads_started":1}
W20260201 03:03:34.976298 29121 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:34.971189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:57939 (local address 127.18.203.254:35875)
0201 03:03:34.971337 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:34.971342 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:34.971411 (+ 69us) server_negotiation.cc:408] Connection header received
0201 03:03:34.971547 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:34.971551 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:34.971601 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:34.971693 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:34.972271 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:34.972769 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:34.973376 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:34.973549 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:34.976186 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:57939: BlockingRecv error: recv got EOF from 127.18.203.195:57939 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Feb 01 03:03:35 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:35 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:35.635568 29124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:35.630725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:41979)
0201 03:03:35.631007 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:35.631021 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:35.631153 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:35.631426 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:35.631428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:35.631645 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:35.631848 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:35.631855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:35.632571 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:35.632575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:35.632966 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:35.632973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:35.633069 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:35.633594 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:35.633608 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:35.635445 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
W20260201 03:03:35.635748 29125 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:35.630800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:41979 (local address 127.18.203.254:35875)
0201 03:03:35.631056 (+ 256us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:35.631060 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:35.631073 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:35.631232 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:35.631234 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:35.631279 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:35.631356 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:35.631949 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:35.632464 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:35.633049 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:35.633183 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:35.635588 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:41979: BlockingRecv error: recv got EOF from 127.18.203.193:41979 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
Feb 01 03:03:35 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:35 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:35.805645 29126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:35.800756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:42435)
0201 03:03:35.801010 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:35.801023 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:35.801131 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:35.801361 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:35.801363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:35.801553 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:35.801747 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:35.801752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:35.802485 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:35.802487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:35.802865 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:35.802871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:35.802969 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:35.803479 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:35.803493 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:35.805519 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
W20260201 03:03:35.805763 29125 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:35.800836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:42435 (local address 127.18.203.254:35875)
0201 03:03:35.801001 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:35.801004 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:35.801086 (+ 82us) server_negotiation.cc:408] Connection header received
0201 03:03:35.801207 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:35.801210 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:35.801263 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:35.801327 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:35.801863 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:35.802385 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:35.802957 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:35.803091 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:35.805651 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:42435: BlockingRecv error: recv got EOF from 127.18.203.194:42435 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Feb 01 03:03:35 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:35 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:35.982113 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:35.977162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:41209)
0201 03:03:35.977459 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:35.977475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:35.977588 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:35.977837 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:35.977840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:35.978051 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:35.978321 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:35.978331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:35.979092 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:35.979095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:35.979486 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:35.979493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:35.979582 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:35.980080 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:35.980093 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:35.981962 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
W20260201 03:03:35.982168 29125 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:35.977280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:41209 (local address 127.18.203.254:35875)
0201 03:03:35.977419 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:35.977424 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:35.977544 (+ 120us) server_negotiation.cc:408] Connection header received
0201 03:03:35.977671 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:35.977674 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:35.977725 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:35.977801 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:35.978434 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:35.978998 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:35.979570 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:35.979708 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:35.982076 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:41209: BlockingRecv error: recv got EOF from 127.18.203.195:41209 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Feb 01 03:03:36 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:36 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:36.641022 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:36.636326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:49383)
0201 03:03:36.636601 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:36.636614 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:36.636705 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:36.636923 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:36.636926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:36.637107 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:36.637319 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:36.637326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:36.638124 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:36.638126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:36.638509 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:36.638515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:36.638599 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:36.639112 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:36.639125 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:36.640903 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":91,"threads_started":1}
W20260201 03:03:36.641175 29129 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:36.636424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:49383 (local address 127.18.203.254:35875)
0201 03:03:36.636644 (+ 220us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:36.636648 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:36.636661 (+ 13us) server_negotiation.cc:408] Connection header received
0201 03:03:36.636779 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:36.636782 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:36.636826 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:36.636904 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:36.637465 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:36.638011 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:36.638636 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:36.638790 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:36.641041 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:49383: BlockingRecv error: recv got EOF from 127.18.203.193:49383 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
Feb 01 03:03:36 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:36 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:36.811642 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:36.806460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:40975)
0201 03:03:36.806821 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:36.806836 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:36.806961 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:36.807252 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:36.807256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:36.807476 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:36.807729 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:36.807738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:36.808531 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:36.808535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:36.808987 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:36.808995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:36.809141 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:36.809674 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:36.809688 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:36.811466 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
W20260201 03:03:36.811741 29129 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:36.806588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:40975 (local address 127.18.203.254:35875)
0201 03:03:36.806735 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:36.806740 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:36.806922 (+ 182us) server_negotiation.cc:408] Connection header received
0201 03:03:36.807074 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:36.807078 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:36.807137 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:36.807215 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:36.807861 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:36.808391 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:36.809113 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:36.809305 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:36.811643 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:40975: BlockingRecv error: recv got EOF from 127.18.203.194:40975 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Feb 01 03:03:36 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:36 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:36.987818 29131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:36.982897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:33675)
0201 03:03:36.983196 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:36.983210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:36.983335 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:36.983595 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:36.983599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:36.983916 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:36.984158 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:36.984167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:36.984934 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:36.984937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:36.985338 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:36.985345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:36.985462 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:36.985979 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:36.985992 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:36.987700 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":101,"threads_started":1}
W20260201 03:03:36.987922 29129 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:36.982968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:33675 (local address 127.18.203.254:35875)
0201 03:03:36.983109 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:36.983114 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:36.983281 (+ 167us) server_negotiation.cc:408] Connection header received
0201 03:03:36.983432 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:36.983436 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:36.983488 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:36.983565 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:36.984294 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:36.984806 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:36.985455 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:36.985635 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:36.987824 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:33675: BlockingRecv error: recv got EOF from 127.18.203.195:33675 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Feb 01 03:03:37 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:37 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:37.646396 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:37.641728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:35317)
0201 03:03:37.642011 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:37.642023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:37.642114 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:37.642361 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:37.642363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:37.642550 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:37.642748 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:37.642754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:37.643530 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:37.643532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:37.643931 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:37.643937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:37.644018 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:37.644547 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:37.644561 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:37.646277 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":97,"threads_started":1}
W20260201 03:03:37.646570 29133 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:37.641839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:35317 (local address 127.18.203.254:35875)
0201 03:03:37.642082 (+ 243us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:37.642086 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:37.642102 (+ 16us) server_negotiation.cc:408] Connection header received
0201 03:03:37.642194 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:37.642196 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:37.642246 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:37.642341 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:37.642851 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:37.643438 (+ 587us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:37.644033 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:37.644227 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:37.646426 (+ 2199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:35317: BlockingRecv error: recv got EOF from 127.18.203.193:35317 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
Feb 01 03:03:37 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:37 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:37.817106 29134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:37.812420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:35477)
0201 03:03:37.812696 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:37.812711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:37.812826 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:37.813061 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:37.813064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:37.813267 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:37.813471 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:37.813478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:37.814182 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:37.814187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:37.814570 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:37.814576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:37.814677 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:37.815218 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:37.815231 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:37.816955 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
W20260201 03:03:37.817210 29133 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:37.812565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:35477 (local address 127.18.203.254:35875)
0201 03:03:37.812698 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:37.812702 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:37.812776 (+ 74us) server_negotiation.cc:408] Connection header received
0201 03:03:37.812908 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:37.812912 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:37.812959 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:37.813024 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:37.813586 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:37.814083 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:37.814674 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:37.814857 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:37.817098 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:35477: BlockingRecv error: recv got EOF from 127.18.203.194:35477 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Feb 01 03:03:37 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:37 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:37.993167 29135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:37.988566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:36367)
0201 03:03:37.988850 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:37.988865 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:37.988976 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:37.989214 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:37.989217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:37.989419 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:37.989616 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:37.989622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:37.990270 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:37.990272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:37.990660 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:37.990666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:37.990751 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:37.991291 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:37.991305 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:37.993018 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
W20260201 03:03:37.993286 29133 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:37.988640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:36367 (local address 127.18.203.254:35875)
0201 03:03:37.988780 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:37.988784 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:37.988935 (+ 151us) server_negotiation.cc:408] Connection header received
0201 03:03:37.989062 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:37.989064 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:37.989112 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:37.989170 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:37.989706 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:37.990189 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:37.990737 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:37.990852 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:37.993150 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:36367: BlockingRecv error: recv got EOF from 127.18.203.195:36367 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Feb 01 03:03:38 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:38 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:38.654991 29136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:38.647215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:52047)
0201 03:03:38.647494 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:38.647508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:38.647642 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:38.647980 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:38.647982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:38.648170 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:38.648381 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:38.648387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:38.649078 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:38.649080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:38.649463 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:38.649468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:38.649550 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:38.653092 (+ 3542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:38.653107 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:38.654865 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
W20260201 03:03:38.655138 29137 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:38.647358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:52047 (local address 127.18.203.254:35875)
0201 03:03:38.647705 (+ 347us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:38.647709 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:38.647725 (+ 16us) server_negotiation.cc:408] Connection header received
0201 03:03:38.647776 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:38.647780 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:38.647839 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:38.647938 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:38.648484 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:38.648992 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:38.649549 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:38.649688 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:38.655015 (+ 5327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:52047: BlockingRecv error: recv got EOF from 127.18.203.193:52047 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":81,"threads_started":1}
Feb 01 03:03:38 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:38 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:38.822746 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:38.817882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:57319)
0201 03:03:38.818169 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:38.818185 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:38.818311 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:38.818556 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:38.818559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:38.818766 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:38.818959 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:38.818966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:38.819737 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:38.819742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:38.820124 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:38.820131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:38.820236 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:38.820809 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:38.820822 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:38.822624 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
W20260201 03:03:38.822893 29137 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:38.818009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:57319 (local address 127.18.203.254:35875)
0201 03:03:38.818169 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:38.818173 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:38.818263 (+ 90us) server_negotiation.cc:408] Connection header received
0201 03:03:38.818399 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:38.818402 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:38.818451 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:38.818521 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:38.819090 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:38.819604 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:38.820239 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:38.820428 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:38.822770 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:57319: BlockingRecv error: recv got EOF from 127.18.203.194:57319 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Feb 01 03:03:38 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:38 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.195@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:38.998479 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:38.993913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.195:33279)
0201 03:03:38.994196 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:38.994210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:38.994318 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:38.994560 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:38.994562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:38.994742 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:38.994939 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:38.994945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:38.995661 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:38.995665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:38.996053 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:38.996059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:38.996153 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:38.996670 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:38.996683 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:38.998358 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
W20260201 03:03:38.998592 29137 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:38.993975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:33279 (local address 127.18.203.254:35875)
0201 03:03:38.994112 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:38.994116 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:38.994277 (+ 161us) server_negotiation.cc:408] Connection header received
0201 03:03:38.994390 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:38.994392 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:38.994439 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:38.994506 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:38.995033 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:38.995524 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:38.996142 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:38.996272 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:38.998478 (+ 2206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.195:33279: BlockingRecv error: recv got EOF from 127.18.203.195:33279 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Feb 01 03:03:39 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:39 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.193@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:39.661213 29140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:39.656069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.193:53351)
0201 03:03:39.656378 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:39.656392 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:39.656488 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:39.656721 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:39.656723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:39.656903 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:39.657108 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:39.657116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:39.658087 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:39.658091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:39.658568 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:39.658578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:39.658681 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:39.659211 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:39.659225 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:39.661022 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
W20260201 03:03:39.661281 29141 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:39.656195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:53351 (local address 127.18.203.254:35875)
0201 03:03:39.656444 (+ 249us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:39.656447 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:39.656461 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:03:39.656572 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:39.656574 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:39.656619 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:39.656695 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:39.657238 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:39.657958 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:39.658681 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:39.658817 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:39.661173 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.193:53351: BlockingRecv error: recv got EOF from 127.18.203.193:53351 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":63,"threads_started":1}
Feb 01 03:03:39 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
Feb 01 03:03:39 dist-test-slave-gf59 krb5kdc[28073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.203.194@KRBTEST.COM for oryx/127.18.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260201 03:03:39.828491 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:39.823565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35875 (local address 127.18.203.194:53213)
0201 03:03:39.823871 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:39.823886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:39.823993 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:39.824210 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:39.824212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:39.824418 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:39.824620 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:39.824626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:39.825356 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:39.825360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:39.825759 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:39.825767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:39.825870 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:39.826393 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:39.826407 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:39.828325 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.203.254:35875: Server oryx/127.18.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":112,"threads_started":1}
W20260201 03:03:39.828584 29141 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:39.823710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:53213 (local address 127.18.203.254:35875)
0201 03:03:39.823856 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:39.823860 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:39.823953 (+ 93us) server_negotiation.cc:408] Connection header received
0201 03:03:39.824068 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:39.824071 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:39.824117 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:39.824191 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:39.824723 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:39.825231 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:39.825849 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:39.825969 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:39.828482 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.203.194:53213: BlockingRecv error: recv got EOF from 127.18.203.194:53213 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260201 03:03:39.913304 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 28665
I20260201 03:03:39.919489 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 28818
I20260201 03:03:39.925032 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 28950
I20260201 03:03:39.930167 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 28595
2026-02-01T03:03:39Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18791 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-02-01T03:03:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:03:39Z Disabled control of system clock
I20260201 03:03:39.961895 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:35641
--webserver_interface=127.18.203.254
--webserver_port=0
--builtin_ntp_servers=127.18.203.212:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:35641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260201 03:03:40.071740 29146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:40.072005 29146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:40.072050 29146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:40.075368 29146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:40.075441 29146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:40.075457 29146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:40.075481 29146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:40.075498 29146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:03:40.079500 29146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:35641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:35641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:40.080695 29146 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:40.081820 29146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:40.087920 29151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:40.088055 29154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:40.088071 29146 server_base.cc:1060] running on GCE node
W20260201 03:03:40.088169 29152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:40.088840 29146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:40.089756 29146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:40.090904 29146 hybrid_clock.cc:648] HybridClock initialized: now 1769915020090894 us; error 30 us; skew 500 ppm
I20260201 03:03:40.092742 29146 webserver.cc:492] Webserver started at http://127.18.203.254:34423/ using document root <none> and password file <none>
I20260201 03:03:40.093315 29146 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:40.093379 29146 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:40.093561 29146 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:40.095396 29146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "36752cef7b8a412a92d4d9edfbd4fab9"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "d1d042566de6151ca87856b3a01347cb"
server_key_iv: "862dc6e236befc160d708c89e3163e53"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.095965 29146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "36752cef7b8a412a92d4d9edfbd4fab9"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "d1d042566de6151ca87856b3a01347cb"
server_key_iv: "862dc6e236befc160d708c89e3163e53"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.099577 29146 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260201 03:03:40.101961 29160 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:40.103101 29146 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:40.103240 29146 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "36752cef7b8a412a92d4d9edfbd4fab9"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "d1d042566de6151ca87856b3a01347cb"
server_key_iv: "862dc6e236befc160d708c89e3163e53"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.103340 29146 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:40.119807 29146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:40.120723 29146 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:40.120965 29146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:40.128846 29212 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:35641 every 8 connection(s)
I20260201 03:03:40.128847 29146 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:35641
I20260201 03:03:40.129967 29146 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:03:40.132923 29213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:40.137637 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 29146
I20260201 03:03:40.137748 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:03:40.137976 19247 external_mini_cluster.cc:1468] Setting key fbfa687c47cc3f3682527c998a396de1
I20260201 03:03:40.138547 29213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9: Bootstrap starting.
I20260201 03:03:40.141250 29213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:40.142187 29213 log.cc:826] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:40.144307 29213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9: No bootstrap required, opened a new log
I20260201 03:03:40.146601 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:40.139257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.18.203.254:35641)
0201 03:03:40.140024 (+ 767us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:40.140035 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:40.140077 (+ 42us) server_negotiation.cc:408] Connection header received
0201 03:03:40.140879 (+ 802us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:40.140908 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:40.141389 (+ 481us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:40.141794 (+ 405us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:40.142361 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.143351 (+ 990us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:40.144111 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.144385 (+ 274us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:40.144762 (+ 377us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:40.144783 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:40.144796 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:03:40.145306 (+ 510us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:40.145364 (+ 58us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:03:40.145384 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:40.145505 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:40.145604 (+ 99us) server_negotiation.cc:300] Negotiation successful
0201 03:03:40.145849 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":164,"threads_started":1}
I20260201 03:03:40.147655 29213 raft_consensus.cc:359] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36752cef7b8a412a92d4d9edfbd4fab9" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35641 } }
I20260201 03:03:40.147864 29213 raft_consensus.cc:385] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:40.147926 29213 raft_consensus.cc:740] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36752cef7b8a412a92d4d9edfbd4fab9, State: Initialized, Role: FOLLOWER
I20260201 03:03:40.148377 29213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36752cef7b8a412a92d4d9edfbd4fab9" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35641 } }
I20260201 03:03:40.148504 29213 raft_consensus.cc:399] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:03:40.148568 29213 raft_consensus.cc:493] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:03:40.148664 29213 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:40.149590 29213 raft_consensus.cc:515] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36752cef7b8a412a92d4d9edfbd4fab9" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35641 } }
I20260201 03:03:40.149906 29213 leader_election.cc:304] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36752cef7b8a412a92d4d9edfbd4fab9; no voters:
I20260201 03:03:40.150178 29213 leader_election.cc:290] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:03:40.150279 29218 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:40.150491 29218 raft_consensus.cc:697] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [term 1 LEADER]: Becoming Leader. State: Replica: 36752cef7b8a412a92d4d9edfbd4fab9, State: Running, Role: LEADER
I20260201 03:03:40.150810 29218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36752cef7b8a412a92d4d9edfbd4fab9" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35641 } }
I20260201 03:03:40.151290 29213 sys_catalog.cc:565] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:40.152385 29220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36752cef7b8a412a92d4d9edfbd4fab9. Latest consensus state: current_term: 1 leader_uuid: "36752cef7b8a412a92d4d9edfbd4fab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36752cef7b8a412a92d4d9edfbd4fab9" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35641 } } }
I20260201 03:03:40.152372 29219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36752cef7b8a412a92d4d9edfbd4fab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36752cef7b8a412a92d4d9edfbd4fab9" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 35641 } } }
I20260201 03:03:40.152515 29219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:40.152515 29220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:40.152889 29227 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:03:40.155776 29227 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:03:40.161068 29227 catalog_manager.cc:1357] Generated new cluster ID: 42de3f62e3cb423c8089d0708a3e5a4b
I20260201 03:03:40.161155 29227 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:03:40.185099 29227 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:03:40.186483 29227 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:03:40.194907 29227 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 36752cef7b8a412a92d4d9edfbd4fab9: Generated new TSK 0
I20260201 03:03:40.195842 29227 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:03:40.200021 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:35641
--builtin_ntp_servers=127.18.203.212:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260201 03:03:40.313604 29237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:40.313990 29237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:40.314101 29237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:40.318166 29237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:40.318243 29237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:40.318339 29237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:03:40.322314 29237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:40.323458 29237 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:40.324685 29237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:40.331319 29242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:40.331341 29243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:40.331501 29237 server_base.cc:1060] running on GCE node
W20260201 03:03:40.331333 29245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:40.331916 29237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:40.332494 29237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:40.333649 29237 hybrid_clock.cc:648] HybridClock initialized: now 1769915020333636 us; error 30 us; skew 500 ppm
I20260201 03:03:40.335460 29237 webserver.cc:492] Webserver started at http://127.18.203.193:43861/ using document root <none> and password file <none>
I20260201 03:03:40.336061 29237 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:40.336123 29237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:40.336314 29237 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:40.337991 29237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "2b8292e334fb4c25b6c0e104a324fc7e"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "15b2809756bdc6e18daa74d53652bcb2"
server_key_iv: "fe718e50ce38866cc78c06835f89f0c6"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.338474 29237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "2b8292e334fb4c25b6c0e104a324fc7e"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "15b2809756bdc6e18daa74d53652bcb2"
server_key_iv: "fe718e50ce38866cc78c06835f89f0c6"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.341779 29237 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:03:40.344071 29251 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:40.345168 29237 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:03:40.345292 29237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "2b8292e334fb4c25b6c0e104a324fc7e"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "15b2809756bdc6e18daa74d53652bcb2"
server_key_iv: "fe718e50ce38866cc78c06835f89f0c6"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.345386 29237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:40.358309 29237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:40.358950 29237 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:40.359125 29237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:40.359683 29237 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:40.360682 29237 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:40.360730 29237 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:40.360781 29237 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:40.360809 29237 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:40.369071 29237 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:36275
I20260201 03:03:40.369122 29364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:36275 every 8 connection(s)
I20260201 03:03:40.370008 29237 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:03:40.376734 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 29237
I20260201 03:03:40.376873 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:03:40.377131 19247 external_mini_cluster.cc:1468] Setting key 3f98aabd7c97eccba7805eff1c789698
I20260201 03:03:40.378086 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:40.371712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:49041 (local address 127.18.203.254:35641)
0201 03:03:40.371836 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:40.371840 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:40.372878 (+ 1038us) server_negotiation.cc:408] Connection header received
0201 03:03:40.373684 (+ 806us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:40.373687 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:40.373741 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:40.373807 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:40.374858 (+ 1051us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.375354 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:40.376084 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.376228 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:40.377597 (+ 1369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:40.377616 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:40.377618 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:03:40.377631 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:40.377657 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:03:40.377666 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:40.377740 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:40.377910 (+ 170us) server_negotiation.cc:300] Negotiation successful
0201 03:03:40.377971 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260201 03:03:40.378818 29367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:40.371998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35641 (local address 127.18.203.193:49041)
0201 03:03:40.372729 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:40.372772 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:40.373493 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:40.374013 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:40.374023 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:40.374107 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:40.374728 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:40.374742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.375485 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:40.375489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:40.375979 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:40.375989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.376181 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:40.377346 (+ 1165us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:03:40.377348 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:40.377382 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:03:40.377436 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:40.377762 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:40.377769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:40.377773 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:03:40.377910 (+ 137us) client_negotiation.cc:240] Negotiation successful
0201 03:03:40.378144 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":128,"threads_started":1}
I20260201 03:03:40.379165 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:35641
--builtin_ntp_servers=127.18.203.212:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260201 03:03:40.379997 29365 heartbeater.cc:344] Connected to a master server at 127.18.203.254:35641
I20260201 03:03:40.380226 29365 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:40.380757 29365 heartbeater.cc:507] Master 127.18.203.254:35641 requested a full tablet report, sending...
I20260201 03:03:40.382261 29177 ts_manager.cc:194] Registered new tserver with Master: 2b8292e334fb4c25b6c0e104a324fc7e (127.18.203.193:36275)
I20260201 03:03:40.383515 29177 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.203.193:49041
W20260201 03:03:40.489799 29368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:40.490056 29368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:40.490130 29368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:40.493505 29368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:40.493577 29368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:40.493673 29368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:03:40.497617 29368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:40.498795 29368 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:40.500002 29368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:40.506631 29376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:40.506670 29373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:40.506626 29374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:40.506942 29368 server_base.cc:1060] running on GCE node
I20260201 03:03:40.507323 29368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:40.507939 29368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:40.509095 29368 hybrid_clock.cc:648] HybridClock initialized: now 1769915020509099 us; error 51 us; skew 500 ppm
I20260201 03:03:40.510813 29368 webserver.cc:492] Webserver started at http://127.18.203.194:35199/ using document root <none> and password file <none>
I20260201 03:03:40.511366 29368 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:40.511430 29368 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:40.511668 29368 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:40.513342 29368 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "8299d925d07c4b20a2096ea12c1f01ec"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "f7903b525d36eb10b83e766deedc08e7"
server_key_iv: "8659addeea23f65855603ff3102de5fb"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.513823 29368 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "8299d925d07c4b20a2096ea12c1f01ec"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "f7903b525d36eb10b83e766deedc08e7"
server_key_iv: "8659addeea23f65855603ff3102de5fb"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.517175 29368 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260201 03:03:40.519423 29382 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:40.520465 29368 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:40.520570 29368 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "8299d925d07c4b20a2096ea12c1f01ec"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "f7903b525d36eb10b83e766deedc08e7"
server_key_iv: "8659addeea23f65855603ff3102de5fb"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.520660 29368 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:40.539104 29368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:40.539886 29368 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:40.540067 29368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:40.540643 29368 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:40.541695 29368 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:40.541747 29368 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:40.541800 29368 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:40.541826 29368 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:40.550930 29368 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:34367
I20260201 03:03:40.550956 29495 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:34367 every 8 connection(s)
I20260201 03:03:40.551990 29368 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:03:40.556434 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 29368
I20260201 03:03:40.556581 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:03:40.556918 19247 external_mini_cluster.cc:1468] Setting key ddba1178771cc13a92145c47c4f622cd
I20260201 03:03:40.559087 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:35641
--builtin_ntp_servers=127.18.203.212:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260201 03:03:40.560938 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:40.553900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:54801 (local address 127.18.203.254:35641)
0201 03:03:40.554058 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:40.554063 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:40.555094 (+ 1031us) server_negotiation.cc:408] Connection header received
0201 03:03:40.556053 (+ 959us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:40.556056 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:40.556127 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:40.556205 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:40.557555 (+ 1350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.558062 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:40.558844 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.559048 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:40.560354 (+ 1306us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:40.560375 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:40.560379 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:03:40.560395 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:40.560420 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:03:40.560430 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:40.560510 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:40.560707 (+ 197us) server_negotiation.cc:300] Negotiation successful
0201 03:03:40.560779 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260201 03:03:40.561852 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:40.554246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35641 (local address 127.18.203.194:54801)
0201 03:03:40.554941 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:40.554980 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:40.555856 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:40.556412 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:40.556425 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:40.556533 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:40.557394 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:40.557410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.558195 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:40.558200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:40.558710 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:40.558720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.558931 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:40.560106 (+ 1175us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:03:40.560108 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:40.560147 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:03:40.560210 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:40.560549 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:40.560557 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:40.560561 (+ 4us) client_negotiation.cc:769] Sending connection context
0201 03:03:40.560705 (+ 144us) client_negotiation.cc:240] Negotiation successful
0201 03:03:40.561005 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":123,"threads_started":1}
I20260201 03:03:40.562925 29496 heartbeater.cc:344] Connected to a master server at 127.18.203.254:35641
I20260201 03:03:40.563177 29496 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:40.563815 29496 heartbeater.cc:507] Master 127.18.203.254:35641 requested a full tablet report, sending...
I20260201 03:03:40.564858 29177 ts_manager.cc:194] Registered new tserver with Master: 8299d925d07c4b20a2096ea12c1f01ec (127.18.203.194:34367)
I20260201 03:03:40.565630 29177 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.203.194:54801
W20260201 03:03:40.671542 29499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:40.671842 29499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:40.671891 29499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:40.675159 29499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:40.675225 29499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:40.675294 29499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:03:40.679152 29499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:40.680320 29499 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:40.681468 29499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:40.688103 29505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:40.688108 29504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:40.688117 29507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:40.688552 29499 server_base.cc:1060] running on GCE node
I20260201 03:03:40.688894 29499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:40.689469 29499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:40.690618 29499 hybrid_clock.cc:648] HybridClock initialized: now 1769915020690604 us; error 37 us; skew 500 ppm
I20260201 03:03:40.692425 29499 webserver.cc:492] Webserver started at http://127.18.203.195:46759/ using document root <none> and password file <none>
I20260201 03:03:40.693004 29499 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:40.693066 29499 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:40.693244 29499 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:40.694974 29499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "339047a6cdd8424a952fcc3c3a1bf662"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "ee3bc21e1806c59efb7cfd5ef6f63c35"
server_key_iv: "8f14c53baa6355393c8beefa15b28b94"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.695483 29499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "339047a6cdd8424a952fcc3c3a1bf662"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "ee3bc21e1806c59efb7cfd5ef6f63c35"
server_key_iv: "8f14c53baa6355393c8beefa15b28b94"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.698829 29499 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260201 03:03:40.701087 29513 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:40.702133 29499 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:40.702245 29499 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "339047a6cdd8424a952fcc3c3a1bf662"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "ee3bc21e1806c59efb7cfd5ef6f63c35"
server_key_iv: "8f14c53baa6355393c8beefa15b28b94"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.702340 29499 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:40.715818 29499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:40.716548 29499 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:40.716732 29499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:40.717298 29499 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:40.718333 29499 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:40.718385 29499 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:40.718446 29499 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:40.718473 29499 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:40.727910 29499 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:42037
I20260201 03:03:40.728080 29626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:42037 every 8 connection(s)
I20260201 03:03:40.728971 29499 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:03:40.736493 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 29499
I20260201 03:03:40.736613 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:03:40.736897 19247 external_mini_cluster.cc:1468] Setting key c411e834322cefb4d156d774dcdc161f
I20260201 03:03:40.738343 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:40.731098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:50921 (local address 127.18.203.254:35641)
0201 03:03:40.731263 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:40.731268 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:40.732770 (+ 1502us) server_negotiation.cc:408] Connection header received
0201 03:03:40.733800 (+ 1030us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:40.733804 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:40.733869 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:40.733947 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:40.735001 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.735521 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:40.736224 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.736444 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:40.737828 (+ 1384us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:40.737844 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:40.737847 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:03:40.737861 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:40.737888 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:03:40.737897 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:40.737979 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:40.738126 (+ 147us) server_negotiation.cc:300] Negotiation successful
0201 03:03:40.738201 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260201 03:03:40.739147 29629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:40.731481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:35641 (local address 127.18.203.195:50921)
0201 03:03:40.732616 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:40.732660 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:40.733557 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:40.734152 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:40.734161 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:40.734243 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:40.734844 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:40.734856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.735668 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:40.735671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:40.736101 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:40.736109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:40.736342 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:40.737585 (+ 1243us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:03:40.737587 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:40.737633 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:03:40.737705 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:40.737980 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:40.737986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:40.737989 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:03:40.738147 (+ 158us) client_negotiation.cc:240] Negotiation successful
0201 03:03:40.738401 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":113,"threads_started":1}
I20260201 03:03:40.740108 29627 heartbeater.cc:344] Connected to a master server at 127.18.203.254:35641
I20260201 03:03:40.740339 29627 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:40.740870 29627 heartbeater.cc:507] Master 127.18.203.254:35641 requested a full tablet report, sending...
I20260201 03:03:40.741796 29177 ts_manager.cc:194] Registered new tserver with Master: 339047a6cdd8424a952fcc3c3a1bf662 (127.18.203.195:42037)
I20260201 03:03:40.742350 29177 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.203.195:50921
I20260201 03:03:40.750761 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260201 03:03:40.753506 29216 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:40.752639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.18.203.254:35641)
0201 03:03:40.752753 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:40.752756 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:40.752896 (+ 140us) server_negotiation.cc:408] Connection header received
0201 03:03:40.752994 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:40.752997 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:40.753039 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:40.753108 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:40.753361 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50610: BlockingRecv error: recv got EOF from 127.0.0.1:50610 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260201 03:03:40.753801 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 29237
I20260201 03:03:40.759763 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 29368
I20260201 03:03:40.765231 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 29499
I20260201 03:03:40.770179 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 29146
2026-02-01T03:03:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (838 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-02-01T03:03:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:03:40Z Disabled control of system clock
I20260201 03:03:40.799695 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:32883
--webserver_interface=127.18.203.254
--webserver_port=0
--builtin_ntp_servers=127.18.203.212:40389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:32883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260201 03:03:40.909231 29638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:40.909495 29638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:40.909548 29638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:40.912885 29638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:40.912961 29638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:40.912977 29638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:40.912995 29638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:40.913012 29638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:03:40.917078 29638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:40389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:32883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:32883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:40.918207 29638 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:40.919349 29638 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:40.925222 29643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:40.925191 29644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:40.925207 29646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:40.925350 29638 server_base.cc:1060] running on GCE node
I20260201 03:03:40.925905 29638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:40.926872 29638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:40.928032 29638 hybrid_clock.cc:648] HybridClock initialized: now 1769915020928019 us; error 48 us; skew 500 ppm
I20260201 03:03:40.930009 29638 webserver.cc:492] Webserver started at http://127.18.203.254:37271/ using document root <none> and password file <none>
I20260201 03:03:40.930608 29638 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:40.930769 29638 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:40.930998 29638 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:40.932823 29638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "309ef7959add4bbf90877c398b942502"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "06f4c90aec008c5e0b91095b3574a934"
server_key_iv: "99b03720841a2dd3db870b4467a55baa"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.933302 29638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "309ef7959add4bbf90877c398b942502"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "06f4c90aec008c5e0b91095b3574a934"
server_key_iv: "99b03720841a2dd3db870b4467a55baa"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.936827 29638 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:03:40.939220 29652 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:40.940300 29638 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260201 03:03:40.940413 29638 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "309ef7959add4bbf90877c398b942502"
format_stamp: "Formatted at 2026-02-01 03:03:40 on dist-test-slave-gf59"
server_key: "06f4c90aec008c5e0b91095b3574a934"
server_key_iv: "99b03720841a2dd3db870b4467a55baa"
server_key_version: "encryptionkey@0"
I20260201 03:03:40.940505 29638 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:40.945369 29638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:40.945924 29638 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:40.946076 29638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:40.953120 29638 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:32883
I20260201 03:03:40.953176 29704 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:32883 every 8 connection(s)
I20260201 03:03:40.954141 29638 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:03:40.955262 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 29638
I20260201 03:03:40.955374 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:03:40.955641 19247 external_mini_cluster.cc:1468] Setting key 2cdee320c62aa67421bb23711f5e831e
I20260201 03:03:40.957234 29705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:40.960978 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:40.956909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.18.203.254:32883)
0201 03:03:40.957551 (+ 642us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:40.957560 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:40.957599 (+ 39us) server_negotiation.cc:408] Connection header received
0201 03:03:40.958207 (+ 608us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:40.958224 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:40.958544 (+ 320us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:40.958828 (+ 284us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:40.959861 (+ 1033us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:40.959866 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:40.959875 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:03:40.959884 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:40.959921 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:03:40.959930 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:40.960042 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:40.960179 (+ 137us) server_negotiation.cc:300] Negotiation successful
0201 03:03:40.960409 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
I20260201 03:03:40.962677 29705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502: Bootstrap starting.
I20260201 03:03:40.964793 29705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:40.965509 29705 log.cc:826] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:40.967231 29705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502: No bootstrap required, opened a new log
I20260201 03:03:40.969702 29705 raft_consensus.cc:359] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309ef7959add4bbf90877c398b942502" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 32883 } }
I20260201 03:03:40.969892 29705 raft_consensus.cc:385] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:40.969949 29705 raft_consensus.cc:740] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309ef7959add4bbf90877c398b942502, State: Initialized, Role: FOLLOWER
I20260201 03:03:40.970345 29705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309ef7959add4bbf90877c398b942502" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 32883 } }
I20260201 03:03:40.970467 29705 raft_consensus.cc:399] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:03:40.970525 29705 raft_consensus.cc:493] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:03:40.970613 29705 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:40.971444 29705 raft_consensus.cc:515] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309ef7959add4bbf90877c398b942502" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 32883 } }
I20260201 03:03:40.971792 29705 leader_election.cc:304] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 309ef7959add4bbf90877c398b942502; no voters:
I20260201 03:03:40.972064 29705 leader_election.cc:290] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:03:40.972154 29710 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:40.972368 29710 raft_consensus.cc:697] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [term 1 LEADER]: Becoming Leader. State: Replica: 309ef7959add4bbf90877c398b942502, State: Running, Role: LEADER
I20260201 03:03:40.972651 29710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309ef7959add4bbf90877c398b942502" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 32883 } }
I20260201 03:03:40.973166 29705 sys_catalog.cc:565] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:40.974303 29711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "309ef7959add4bbf90877c398b942502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309ef7959add4bbf90877c398b942502" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 32883 } } }
I20260201 03:03:40.974300 29712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 309ef7959add4bbf90877c398b942502. Latest consensus state: current_term: 1 leader_uuid: "309ef7959add4bbf90877c398b942502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309ef7959add4bbf90877c398b942502" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 32883 } } }
I20260201 03:03:40.974442 29711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:40.974447 29712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502 [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:40.974802 29719 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:03:40.977327 29719 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:03:40.982367 29719 catalog_manager.cc:1357] Generated new cluster ID: 9e53067cf4784204863d2eb63960ff0f
I20260201 03:03:40.982445 29719 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:03:40.990633 29719 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:03:40.990940 29719 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:03:40.998072 29719 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 309ef7959add4bbf90877c398b942502: Generated new TSK 0
I20260201 03:03:40.998785 29719 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260201 03:03:41.007795 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:32883
--builtin_ntp_servers=127.18.203.212:40389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260201 03:03:41.117152 29729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:41.117419 29729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:41.117479 29729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:41.120831 29729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:41.120906 29729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:41.120988 29729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:03:41.124958 29729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:40389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:32883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:41.126096 29729 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:41.127260 29729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:41.134021 29734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:41.134053 29737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:41.134037 29735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:41.134656 29729 server_base.cc:1060] running on GCE node
I20260201 03:03:41.135083 29729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:41.135823 29729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:41.136996 29729 hybrid_clock.cc:648] HybridClock initialized: now 1769915021136984 us; error 30 us; skew 500 ppm
I20260201 03:03:41.139329 29729 webserver.cc:492] Webserver started at http://127.18.203.193:41947/ using document root <none> and password file <none>
I20260201 03:03:41.140074 29729 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:41.140163 29729 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:41.140380 29729 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:41.142644 29729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "673e3c7ded1241f8aceb3a25b964346b"
format_stamp: "Formatted at 2026-02-01 03:03:41 on dist-test-slave-gf59"
server_key: "0422cc7c821e6ab33d8128fa0f33628d"
server_key_iv: "a5f61681626e1b7f9554a58e226c9311"
server_key_version: "encryptionkey@0"
I20260201 03:03:41.143173 29729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "673e3c7ded1241f8aceb3a25b964346b"
format_stamp: "Formatted at 2026-02-01 03:03:41 on dist-test-slave-gf59"
server_key: "0422cc7c821e6ab33d8128fa0f33628d"
server_key_iv: "a5f61681626e1b7f9554a58e226c9311"
server_key_version: "encryptionkey@0"
I20260201 03:03:41.146720 29729 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260201 03:03:41.149017 29743 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:41.150072 29729 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:41.150198 29729 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "673e3c7ded1241f8aceb3a25b964346b"
format_stamp: "Formatted at 2026-02-01 03:03:41 on dist-test-slave-gf59"
server_key: "0422cc7c821e6ab33d8128fa0f33628d"
server_key_iv: "a5f61681626e1b7f9554a58e226c9311"
server_key_version: "encryptionkey@0"
I20260201 03:03:41.150302 29729 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:41.155300 29729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:41.155947 29729 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:41.156112 29729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:41.156643 29729 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:41.157617 29729 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:41.157672 29729 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:41.157727 29729 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:41.157754 29729 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:41.166338 29729 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:43465
I20260201 03:03:41.166386 29856 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:43465 every 8 connection(s)
I20260201 03:03:41.167340 29729 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:03:41.172447 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:41.169062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:55871 (local address 127.18.203.254:32883)
0201 03:03:41.169211 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:41.169215 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:41.170114 (+ 899us) server_negotiation.cc:408] Connection header received
0201 03:03:41.170891 (+ 777us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:41.170893 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:41.170935 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:41.171008 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:41.172049 (+ 1041us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:41.172052 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:41.172055 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:03:41.172066 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:41.172089 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:03:41.172097 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:41.172172 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:41.172297 (+ 125us) server_negotiation.cc:300] Negotiation successful
0201 03:03:41.172333 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260201 03:03:41.173187 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:41.169349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:32883 (local address 127.18.203.193:55871)
0201 03:03:41.169979 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:41.170013 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:41.170720 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:41.171175 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:41.171183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:41.171268 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:41.171810 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:03:41.171812 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:41.171847 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:03:41.171904 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:41.172182 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:41.172187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:41.172190 (+ 3us) client_negotiation.cc:769] Sending connection context
0201 03:03:41.172355 (+ 165us) client_negotiation.cc:240] Negotiation successful
0201 03:03:41.172548 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":101,"threads_started":1}
I20260201 03:03:41.174289 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 29729
I20260201 03:03:41.174328 29857 heartbeater.cc:344] Connected to a master server at 127.18.203.254:32883
I20260201 03:03:41.174423 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:03:41.174571 29857 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:41.174711 19247 external_mini_cluster.cc:1468] Setting key 2e08e656a834409917ab02d0251948a7
I20260201 03:03:41.175012 29857 heartbeater.cc:507] Master 127.18.203.254:32883 requested a full tablet report, sending...
I20260201 03:03:41.176574 29669 ts_manager.cc:194] Registered new tserver with Master: 673e3c7ded1241f8aceb3a25b964346b (127.18.203.193:43465)
I20260201 03:03:41.176761 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:32883
--builtin_ntp_servers=127.18.203.212:40389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260201 03:03:41.286216 29860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:41.286468 29860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:41.286520 29860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:41.289700 29860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:41.289755 29860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:41.289829 29860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:03:41.293598 29860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:40389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:32883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:41.294687 29860 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:41.295800 29860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:41.302255 29868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:41.302278 29866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:41.302278 29865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:41.302592 29860 server_base.cc:1060] running on GCE node
I20260201 03:03:41.302940 29860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:41.303515 29860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:41.304662 29860 hybrid_clock.cc:648] HybridClock initialized: now 1769915021304650 us; error 30 us; skew 500 ppm
I20260201 03:03:41.306453 29860 webserver.cc:492] Webserver started at http://127.18.203.194:37195/ using document root <none> and password file <none>
I20260201 03:03:41.306993 29860 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:41.307050 29860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:41.307250 29860 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:41.309000 29860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "810b5d750857438c90cb0f8e7f9acf72"
format_stamp: "Formatted at 2026-02-01 03:03:41 on dist-test-slave-gf59"
server_key: "b6847650ff8a2726f4c1ce472e0fd365"
server_key_iv: "0cbbcc2a0cdaa2b4017ac0d523a17b1d"
server_key_version: "encryptionkey@0"
I20260201 03:03:41.309471 29860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "810b5d750857438c90cb0f8e7f9acf72"
format_stamp: "Formatted at 2026-02-01 03:03:41 on dist-test-slave-gf59"
server_key: "b6847650ff8a2726f4c1ce472e0fd365"
server_key_iv: "0cbbcc2a0cdaa2b4017ac0d523a17b1d"
server_key_version: "encryptionkey@0"
I20260201 03:03:41.312837 29860 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260201 03:03:41.315168 29874 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:41.316253 29860 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:41.316361 29860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "810b5d750857438c90cb0f8e7f9acf72"
format_stamp: "Formatted at 2026-02-01 03:03:41 on dist-test-slave-gf59"
server_key: "b6847650ff8a2726f4c1ce472e0fd365"
server_key_iv: "0cbbcc2a0cdaa2b4017ac0d523a17b1d"
server_key_version: "encryptionkey@0"
I20260201 03:03:41.316465 29860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:41.321569 29860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:41.322168 29860 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:41.322338 29860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:41.322867 29860 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:41.323931 29860 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:41.323984 29860 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:41.324043 29860 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:41.324076 29860 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:41.333503 29860 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:34913
I20260201 03:03:41.333542 29987 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:34913 every 8 connection(s)
I20260201 03:03:41.334514 29860 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:03:41.339906 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:41.336272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:36551 (local address 127.18.203.254:32883)
0201 03:03:41.336445 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:41.336449 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:41.337374 (+ 925us) server_negotiation.cc:408] Connection header received
0201 03:03:41.338191 (+ 817us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:41.338196 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:41.338247 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:41.338339 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:41.339377 (+ 1038us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:41.339382 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:41.339385 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:03:41.339400 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:41.339425 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:03:41.339435 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:41.339520 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:41.339696 (+ 176us) server_negotiation.cc:300] Negotiation successful
0201 03:03:41.339747 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260201 03:03:41.340610 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:41.336550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:32883 (local address 127.18.203.194:36551)
0201 03:03:41.337221 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:41.337258 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:41.337986 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:41.338516 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:41.338525 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:41.338611 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:41.339136 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:03:41.339138 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:41.339171 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:03:41.339228 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:41.339522 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:41.339532 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:41.339537 (+ 5us) client_negotiation.cc:769] Sending connection context
0201 03:03:41.339763 (+ 226us) client_negotiation.cc:240] Negotiation successful
0201 03:03:41.339976 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":87,"threads_started":1}
I20260201 03:03:41.341586 29988 heartbeater.cc:344] Connected to a master server at 127.18.203.254:32883
I20260201 03:03:41.341821 29988 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:41.342232 29988 heartbeater.cc:507] Master 127.18.203.254:32883 requested a full tablet report, sending...
I20260201 03:03:41.343181 29669 ts_manager.cc:194] Registered new tserver with Master: 810b5d750857438c90cb0f8e7f9acf72 (127.18.203.194:34913)
I20260201 03:03:41.343665 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 29860
I20260201 03:03:41.343779 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:03:41.344077 19247 external_mini_cluster.cc:1468] Setting key 9cae5c7ad5a00d0cdeebe46d0425f94f
I20260201 03:03:41.345885 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:32883
--builtin_ntp_servers=127.18.203.212:40389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260201 03:03:41.451493 29991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:41.451756 29991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:41.451818 29991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:41.455075 29991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:41.455142 29991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:41.455233 29991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:03:41.459055 29991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:40389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:32883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:41.460260 29991 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:41.461476 29991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:41.468003 29997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:41.468008 29996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:41.468089 29999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:41.468302 29991 server_base.cc:1060] running on GCE node
I20260201 03:03:41.468729 29991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:41.469286 29991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:41.470427 29991 hybrid_clock.cc:648] HybridClock initialized: now 1769915021470415 us; error 33 us; skew 500 ppm
I20260201 03:03:41.472223 29991 webserver.cc:492] Webserver started at http://127.18.203.195:35297/ using document root <none> and password file <none>
I20260201 03:03:41.472762 29991 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:41.472816 29991 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:41.472996 29991 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:41.474743 29991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "8f9d8db007f8442590ae3b685e12b3b9"
format_stamp: "Formatted at 2026-02-01 03:03:41 on dist-test-slave-gf59"
server_key: "558b0a5fb4eeb1c75f9b04b95e8123f5"
server_key_iv: "31f25351afe620801121ac06f81a8f89"
server_key_version: "encryptionkey@0"
I20260201 03:03:41.475243 29991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "8f9d8db007f8442590ae3b685e12b3b9"
format_stamp: "Formatted at 2026-02-01 03:03:41 on dist-test-slave-gf59"
server_key: "558b0a5fb4eeb1c75f9b04b95e8123f5"
server_key_iv: "31f25351afe620801121ac06f81a8f89"
server_key_version: "encryptionkey@0"
I20260201 03:03:41.478578 29991 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260201 03:03:41.480758 30005 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:41.481729 29991 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:41.481824 29991 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "8f9d8db007f8442590ae3b685e12b3b9"
format_stamp: "Formatted at 2026-02-01 03:03:41 on dist-test-slave-gf59"
server_key: "558b0a5fb4eeb1c75f9b04b95e8123f5"
server_key_iv: "31f25351afe620801121ac06f81a8f89"
server_key_version: "encryptionkey@0"
I20260201 03:03:41.481915 29991 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:41.486645 29991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:41.487185 29991 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:41.487341 29991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:41.487860 29991 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:41.488772 29991 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:41.488817 29991 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:41.488857 29991 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:41.488885 29991 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:41.497646 29991 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:41533
I20260201 03:03:41.497694 30118 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:41533 every 8 connection(s)
I20260201 03:03:41.498558 29991 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:03:41.502058 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 29991
I20260201 03:03:41.502195 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:03:41.502480 19247 external_mini_cluster.cc:1468] Setting key 7fa120759ec49bed75b12e9374ab09df
I20260201 03:03:41.503985 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:41.500309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:55999 (local address 127.18.203.254:32883)
0201 03:03:41.500430 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:41.500434 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:41.501455 (+ 1021us) server_negotiation.cc:408] Connection header received
0201 03:03:41.502274 (+ 819us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:41.502279 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:41.502327 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:41.502429 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:41.503471 (+ 1042us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:41.503475 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:41.503478 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0201 03:03:41.503490 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:41.503512 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0201 03:03:41.503523 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:41.503639 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:41.503813 (+ 174us) server_negotiation.cc:300] Negotiation successful
0201 03:03:41.503859 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260201 03:03:41.504701 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:41.500578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:32883 (local address 127.18.203.195:55999)
0201 03:03:41.501304 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:41.501350 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:41.502077 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:41.502581 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:41.502591 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:41.502675 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:41.503228 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0201 03:03:41.503230 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:41.503274 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0201 03:03:41.503343 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:41.503650 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:41.503657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:41.503662 (+ 5us) client_negotiation.cc:769] Sending connection context
0201 03:03:41.503854 (+ 192us) client_negotiation.cc:240] Negotiation successful
0201 03:03:41.504021 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20260201 03:03:41.505635 30119 heartbeater.cc:344] Connected to a master server at 127.18.203.254:32883
I20260201 03:03:41.505856 30119 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:41.506235 30119 heartbeater.cc:507] Master 127.18.203.254:32883 requested a full tablet report, sending...
I20260201 03:03:41.507104 29669 ts_manager.cc:194] Registered new tserver with Master: 8f9d8db007f8442590ae3b685e12b3b9 (127.18.203.195:41533)
I20260201 03:03:41.516371 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260201 03:03:41.518963 29708 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:03:41.518142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.18.203.254:32883)
0201 03:03:41.518273 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:41.518278 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:41.518405 (+ 127us) server_negotiation.cc:408] Connection header received
0201 03:03:41.518546 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:41.518548 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:41.518587 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:41.518648 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:41.518859 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40370: BlockingRecv error: recv got EOF from 127.0.0.1:40370 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260201 03:03:41.519423 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 29729
I20260201 03:03:41.525481 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 29860
I20260201 03:03:41.530340 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 29991
I20260201 03:03:41.535353 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 29638
2026-02-01T03:03:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (764 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:03:41 dist-test-slave-gf59 krb5kdc[30129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:03:41 dist-test-slave-gf59 krb5kdc[30129](info): set up 2 sockets
Feb 01 03:03:41 dist-test-slave-gf59 krb5kdc[30129](info): commencing operation
krb5kdc: starting...
W20260201 03:03:43.380311 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.824s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:03:43 dist-test-slave-gf59 krb5kdc[30129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:03:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:03:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:43.524968 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:39937
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:39937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:43.637225 30145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:43.637487 30145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:43.637547 30145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:43.640889 30145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:03:43.640957 30145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:43.640980 30145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:43.640998 30145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:03:43.641016 30145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:03:43.645434 30145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:39937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:39937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:43.646553 30145 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:43.647416 30145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:43.653096 30153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:43.653091 30151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:43.653093 30150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:43.653462 30145 server_base.cc:1060] running on GCE node
I20260201 03:03:43.653954 30145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:43.654870 30145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:43.656003 30145 hybrid_clock.cc:648] HybridClock initialized: now 1769915023655983 us; error 37 us; skew 500 ppm
Feb 01 03:03:43 dist-test-slave-gf59 krb5kdc[30129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915023, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:43.658754 30145 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:03:43.659942 30145 webserver.cc:492] Webserver started at http://127.18.203.254:38135/ using document root <none> and password file <none>
I20260201 03:03:43.660494 30145 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:43.660547 30145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:43.660729 30145 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:43.662622 30145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "c4731cabcf764559b259013bba5733cf"
format_stamp: "Formatted at 2026-02-01 03:03:43 on dist-test-slave-gf59"
server_key: "1030248146ca9975fec3395b0f7e8d8d"
server_key_iv: "8b894f4e37f8a6695fe1fab3004007f1"
server_key_version: "encryptionkey@0"
I20260201 03:03:43.663169 30145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "c4731cabcf764559b259013bba5733cf"
format_stamp: "Formatted at 2026-02-01 03:03:43 on dist-test-slave-gf59"
server_key: "1030248146ca9975fec3395b0f7e8d8d"
server_key_iv: "8b894f4e37f8a6695fe1fab3004007f1"
server_key_version: "encryptionkey@0"
I20260201 03:03:43.666791 30145 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260201 03:03:43.669277 30160 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:43.670413 30145 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:43.670544 30145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "c4731cabcf764559b259013bba5733cf"
format_stamp: "Formatted at 2026-02-01 03:03:43 on dist-test-slave-gf59"
server_key: "1030248146ca9975fec3395b0f7e8d8d"
server_key_iv: "8b894f4e37f8a6695fe1fab3004007f1"
server_key_version: "encryptionkey@0"
I20260201 03:03:43.670652 30145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:43.690284 30145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:43.697667 30145 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:43.697873 30145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:43.705137 30145 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:39937
I20260201 03:03:43.705137 30212 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:39937 every 8 connection(s)
I20260201 03:03:43.706190 30145 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:03:43.709055 30213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:43.711869 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 30145
I20260201 03:03:43.711992 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:03:43.712227 19247 external_mini_cluster.cc:1468] Setting key 3a1a0eab6ce0b35fd4e913712554a7a7
I20260201 03:03:43.714928 30213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf: Bootstrap starting.
I20260201 03:03:43.717337 30213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:43.718149 30213 log.cc:826] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf: Log is configured to *not* fsync() on all Append() calls
Feb 01 03:03:43 dist-test-slave-gf59 krb5kdc[30129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:03:43.720108 30213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf: No bootstrap required, opened a new log
I20260201 03:03:43.722666 30213 raft_consensus.cc:359] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4731cabcf764559b259013bba5733cf" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 39937 } }
I20260201 03:03:43.722851 30213 raft_consensus.cc:385] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:43.722909 30213 raft_consensus.cc:740] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4731cabcf764559b259013bba5733cf, State: Initialized, Role: FOLLOWER
I20260201 03:03:43.723455 30213 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4731cabcf764559b259013bba5733cf" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 39937 } }
I20260201 03:03:43.723598 30213 raft_consensus.cc:399] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:03:43.723683 30213 raft_consensus.cc:493] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:03:43.723767 30213 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:43.724310 30216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:43.713429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.18.203.254:39937)
0201 03:03:43.713827 (+ 398us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:43.713837 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:43.713873 (+ 36us) server_negotiation.cc:408] Connection header received
0201 03:03:43.714394 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:43.714412 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:43.714717 (+ 305us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:43.714993 (+ 276us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:43.716052 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:43.716798 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:43.717636 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:43.717832 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:43.720141 (+ 2309us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:43.720158 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:43.720169 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:43.720215 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:43.721822 (+ 1607us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:43.722258 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:43.722262 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:43.722266 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:43.722334 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:43.722542 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:43.722545 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:43.722546 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:43.722855 (+ 309us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:43.723059 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:43.723319 (+ 260us) server_negotiation.cc:300] Negotiation successful
0201 03:03:43.723567 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":119,"threads_started":1}
I20260201 03:03:43.724679 30213 raft_consensus.cc:515] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4731cabcf764559b259013bba5733cf" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 39937 } }
I20260201 03:03:43.725004 30213 leader_election.cc:304] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4731cabcf764559b259013bba5733cf; no voters:
I20260201 03:03:43.725273 30213 leader_election.cc:290] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:03:43.725356 30218 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:43.725515 30218 raft_consensus.cc:697] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [term 1 LEADER]: Becoming Leader. State: Replica: c4731cabcf764559b259013bba5733cf, State: Running, Role: LEADER
I20260201 03:03:43.725845 30218 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4731cabcf764559b259013bba5733cf" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 39937 } }
I20260201 03:03:43.726413 30213 sys_catalog.cc:565] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:03:43.727507 30219 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4731cabcf764559b259013bba5733cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4731cabcf764559b259013bba5733cf" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 39937 } } }
I20260201 03:03:43.727733 30219 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:43.727985 30220 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4731cabcf764559b259013bba5733cf. Latest consensus state: current_term: 1 leader_uuid: "c4731cabcf764559b259013bba5733cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4731cabcf764559b259013bba5733cf" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 39937 } } }
I20260201 03:03:43.728124 30227 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:03:43.728127 30220 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf [sys.catalog]: This master's current role is: LEADER
I20260201 03:03:43.731105 30227 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:03:43.736788 30227 catalog_manager.cc:1357] Generated new cluster ID: 9b02bdbaab3b4a1e869c5bb02789c3a4
I20260201 03:03:43.736878 30227 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:03:43.744619 30227 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:03:43.745591 30227 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:03:43.752491 30227 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf: Generated new TSK 0
I20260201 03:03:43.753445 30227 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:43.812825 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:39937
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:43.923784 30241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:43.924057 30241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:43.924130 30241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:43.927445 30241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:43.927526 30241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:43.927659 30241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:03:43.932027 30241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:39937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:43.933203 30241 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:43.934126 30241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:43.940866 30249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:43.940831 30247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:43.940956 30241 server_base.cc:1060] running on GCE node
W20260201 03:03:43.940819 30246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:43.941437 30241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:43.941988 30241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:43.943135 30241 hybrid_clock.cc:648] HybridClock initialized: now 1769915023943122 us; error 42 us; skew 500 ppm
Feb 01 03:03:43 dist-test-slave-gf59 krb5kdc[30129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915023, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:43.945739 30241 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:03:43.946821 30241 webserver.cc:492] Webserver started at http://127.18.203.193:34617/ using document root <none> and password file <none>
I20260201 03:03:43.947372 30241 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:43.947422 30241 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:43.947604 30241 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:43.949406 30241 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "34d472685386498ea2c5e7dbd8672de2"
format_stamp: "Formatted at 2026-02-01 03:03:43 on dist-test-slave-gf59"
server_key: "9018951bf2a71a73a72932e1bf990320"
server_key_iv: "45d0e7943140c913b52e5746630c0939"
server_key_version: "encryptionkey@0"
I20260201 03:03:43.949868 30241 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "34d472685386498ea2c5e7dbd8672de2"
format_stamp: "Formatted at 2026-02-01 03:03:43 on dist-test-slave-gf59"
server_key: "9018951bf2a71a73a72932e1bf990320"
server_key_iv: "45d0e7943140c913b52e5746630c0939"
server_key_version: "encryptionkey@0"
I20260201 03:03:43.953188 30241 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:03:43.955291 30256 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:43.956300 30241 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260201 03:03:43.956401 30241 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "34d472685386498ea2c5e7dbd8672de2"
format_stamp: "Formatted at 2026-02-01 03:03:43 on dist-test-slave-gf59"
server_key: "9018951bf2a71a73a72932e1bf990320"
server_key_iv: "45d0e7943140c913b52e5746630c0939"
server_key_version: "encryptionkey@0"
I20260201 03:03:43.956487 30241 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:43.973264 30241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:43.976315 30241 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:43.976512 30241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:43.977077 30241 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:43.977980 30241 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:43.978029 30241 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:43.978081 30241 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:43.978106 30241 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:43.986949 30241 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:37405
I20260201 03:03:43.986977 30369 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:37405 every 8 connection(s)
I20260201 03:03:43.987917 30241 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:03:43.989840 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 30241
I20260201 03:03:43.989969 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:03:43.990269 19247 external_mini_cluster.cc:1468] Setting key ba32bf31d88d30598d0318cb95b3290a
Feb 01 03:03:43 dist-test-slave-gf59 krb5kdc[30129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915023, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
I20260201 03:03:44.001312 30216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:43.989600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:33655 (local address 127.18.203.254:39937)
0201 03:03:43.989773 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:43.989777 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:43.990485 (+ 708us) server_negotiation.cc:408] Connection header received
0201 03:03:43.991428 (+ 943us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:43.991431 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:43.991489 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:43.991561 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:43.993869 (+ 2308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:43.994624 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:43.995293 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:43.995495 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:43.997775 (+ 2280us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:43.997801 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:43.997804 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:43.997839 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:43.999368 (+ 1529us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:43.999884 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:43.999889 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:43.999891 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:43.999947 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:44.000216 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:44.000220 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:44.000221 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:44.000428 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:44.000527 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:44.001052 (+ 525us) server_negotiation.cc:300] Negotiation successful
0201 03:03:44.001166 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260201 03:03:44.001997 30372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:43.989833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:39937 (local address 127.18.203.193:33655)
0201 03:03:43.990327 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:43.990374 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:43.991233 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:43.991943 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:43.991954 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:43.992485 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:43.993729 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:43.993747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:43.994753 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:43.994756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:43.995175 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:43.995182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:43.995399 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:43.996006 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:43.996034 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:43.997617 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:43.999488 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:43.999493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:43.999503 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:43.999787 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:44.000035 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:44.000037 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:44.000038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:44.000122 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:44.000516 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:44.000522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:44.000791 (+ 269us) client_negotiation.cc:769] Sending connection context
0201 03:03:44.000971 (+ 180us) client_negotiation.cc:240] Negotiation successful
0201 03:03:44.001214 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
I20260201 03:03:44.003279 30370 heartbeater.cc:344] Connected to a master server at 127.18.203.254:39937
I20260201 03:03:44.003542 30370 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:44.004109 30370 heartbeater.cc:507] Master 127.18.203.254:39937 requested a full tablet report, sending...
I20260201 03:03:44.005613 30177 ts_manager.cc:194] Registered new tserver with Master: 34d472685386498ea2c5e7dbd8672de2 (127.18.203.193:37405)
I20260201 03:03:44.006793 30177 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:33655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:44.039912 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:39937
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:44.147739 30377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:44.147993 30377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:44.148065 30377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:44.151355 30377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:44.151440 30377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:44.151553 30377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:03:44.155864 30377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:39937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:44.156898 30377 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:44.157786 30377 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:44.164397 30383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:44.164419 30382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:44.164575 30377 server_base.cc:1060] running on GCE node
W20260201 03:03:44.164387 30385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:44.165062 30377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:44.165621 30377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:44.166774 30377 hybrid_clock.cc:648] HybridClock initialized: now 1769915024166764 us; error 29 us; skew 500 ppm
Feb 01 03:03:44 dist-test-slave-gf59 krb5kdc[30129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915024, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:44.169461 30377 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:03:44.170634 30377 webserver.cc:492] Webserver started at http://127.18.203.194:43269/ using document root <none> and password file <none>
I20260201 03:03:44.171201 30377 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:44.171255 30377 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:44.171452 30377 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:44.173233 30377 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "c8364859d2044a7896189d8a0c502d0d"
format_stamp: "Formatted at 2026-02-01 03:03:44 on dist-test-slave-gf59"
server_key: "9b9395e64276cebdbaaa26f2ef190be7"
server_key_iv: "68a53d43ad85557cd584c04dd9848321"
server_key_version: "encryptionkey@0"
I20260201 03:03:44.173727 30377 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "c8364859d2044a7896189d8a0c502d0d"
format_stamp: "Formatted at 2026-02-01 03:03:44 on dist-test-slave-gf59"
server_key: "9b9395e64276cebdbaaa26f2ef190be7"
server_key_iv: "68a53d43ad85557cd584c04dd9848321"
server_key_version: "encryptionkey@0"
I20260201 03:03:44.177222 30377 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260201 03:03:44.179309 30392 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:44.180292 30377 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:03:44.180406 30377 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "c8364859d2044a7896189d8a0c502d0d"
format_stamp: "Formatted at 2026-02-01 03:03:44 on dist-test-slave-gf59"
server_key: "9b9395e64276cebdbaaa26f2ef190be7"
server_key_iv: "68a53d43ad85557cd584c04dd9848321"
server_key_version: "encryptionkey@0"
I20260201 03:03:44.180491 30377 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:44.217471 30377 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:44.220355 30377 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:44.220546 30377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:44.221076 30377 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:44.221943 30377 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:44.221992 30377 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:44.222043 30377 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:44.222069 30377 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:44.231029 30377 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:33337
I20260201 03:03:44.231055 30505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:33337 every 8 connection(s)
I20260201 03:03:44.232136 30377 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:03:44.235800 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 30377
I20260201 03:03:44.235934 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:03:44.236207 19247 external_mini_cluster.cc:1468] Setting key b1b9bfcc685ce49790800cd8c53321cd
Feb 01 03:03:44 dist-test-slave-gf59 krb5kdc[30129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915024, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:03:44.244740 30216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.233853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:60363 (local address 127.18.203.254:39937)
0201 03:03:44.233977 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:44.233981 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:44.234724 (+ 743us) server_negotiation.cc:408] Connection header received
0201 03:03:44.235535 (+ 811us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:44.235539 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:44.235590 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:44.235697 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:44.237776 (+ 2079us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.238277 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.239024 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.239165 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.241465 (+ 2300us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:44.241484 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:44.241486 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:44.241511 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:44.242876 (+ 1365us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:44.243347 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:44.243350 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:44.243352 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:44.243396 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:44.243683 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:44.243687 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:44.243689 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:44.243844 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:44.243932 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:44.244435 (+ 503us) server_negotiation.cc:300] Negotiation successful
0201 03:03:44.244574 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260201 03:03:44.245393 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.234149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:39937 (local address 127.18.203.194:60363)
0201 03:03:44.234561 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:44.234601 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:44.235340 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:44.235892 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:44.235905 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:44.236485 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:44.237634 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.237654 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.238399 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.238402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:44.238919 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.238930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.239156 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.239785 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:44.239825 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:44.241312 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:44.243000 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:44.243006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:44.243016 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:44.243252 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:44.243481 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:44.243483 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:44.243485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:44.243577 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:44.243944 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:44.243951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:44.244197 (+ 246us) client_negotiation.cc:769] Sending connection context
0201 03:03:44.244387 (+ 190us) client_negotiation.cc:240] Negotiation successful
0201 03:03:44.244672 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
I20260201 03:03:44.246492 30506 heartbeater.cc:344] Connected to a master server at 127.18.203.254:39937
I20260201 03:03:44.246735 30506 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:44.247243 30506 heartbeater.cc:507] Master 127.18.203.254:39937 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
I20260201 03:03:44.248296 30177 ts_manager.cc:194] Registered new tserver with Master: c8364859d2044a7896189d8a0c502d0d (127.18.203.194:33337)
I20260201 03:03:44.248845 30177 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:60363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:03:44.286212 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:39937
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:03:44.393712 30513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:03:44.393975 30513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:03:44.394057 30513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:03:44.397396 30513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:03:44.397475 30513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:03:44.397576 30513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:03:44.401876 30513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:39937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:03:44.403008 30513 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:03:44.403940 30513 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:03:44.410516 30521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:44.410524 30519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:03:44.410529 30518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:03:44.410809 30513 server_base.cc:1060] running on GCE node
I20260201 03:03:44.411160 30513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:03:44.411778 30513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:03:44.412927 30513 hybrid_clock.cc:648] HybridClock initialized: now 1769915024412913 us; error 34 us; skew 500 ppm
Feb 01 03:03:44 dist-test-slave-gf59 krb5kdc[30129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915024, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:03:44.415551 30513 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:03:44.416615 30513 webserver.cc:492] Webserver started at http://127.18.203.195:41971/ using document root <none> and password file <none>
I20260201 03:03:44.417140 30513 fs_manager.cc:362] Metadata directory not provided
I20260201 03:03:44.417191 30513 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:03:44.417367 30513 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260201 03:03:44.419030 30513 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "4ac9809d418b466cb0237fe09b74f147"
format_stamp: "Formatted at 2026-02-01 03:03:44 on dist-test-slave-gf59"
server_key: "6b86495b874192ffa746ec72a16b1014"
server_key_iv: "cd25c30f985eb2ede37f1d0fb4ed4826"
server_key_version: "encryptionkey@0"
I20260201 03:03:44.419510 30513 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "4ac9809d418b466cb0237fe09b74f147"
format_stamp: "Formatted at 2026-02-01 03:03:44 on dist-test-slave-gf59"
server_key: "6b86495b874192ffa746ec72a16b1014"
server_key_iv: "cd25c30f985eb2ede37f1d0fb4ed4826"
server_key_version: "encryptionkey@0"
I20260201 03:03:44.422750 30513 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260201 03:03:44.424889 30528 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:44.425923 30513 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260201 03:03:44.426041 30513 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "4ac9809d418b466cb0237fe09b74f147"
format_stamp: "Formatted at 2026-02-01 03:03:44 on dist-test-slave-gf59"
server_key: "6b86495b874192ffa746ec72a16b1014"
server_key_iv: "cd25c30f985eb2ede37f1d0fb4ed4826"
server_key_version: "encryptionkey@0"
I20260201 03:03:44.426138 30513 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:03:44.440523 30513 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:03:44.443068 30513 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:03:44.443253 30513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:03:44.443837 30513 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:03:44.444718 30513 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:03:44.444767 30513 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:44.444819 30513 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:03:44.444849 30513 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:03:44.453327 30513 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:35491
I20260201 03:03:44.453354 30641 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:35491 every 8 connection(s)
I20260201 03:03:44.454269 30513 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:03:44.462389 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 30513
Feb 01 03:03:44 dist-test-slave-gf59 krb5kdc[30129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915024, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:03:44.462509 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:03:44.462735 19247 external_mini_cluster.cc:1468] Setting key 41ac6371ad6bb8d58d6cc6588b413a3e
I20260201 03:03:44.466603 30216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.455948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:57889 (local address 127.18.203.254:39937)
0201 03:03:44.456065 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:44.456069 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:44.456748 (+ 679us) server_negotiation.cc:408] Connection header received
0201 03:03:44.457599 (+ 851us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:44.457603 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:44.457663 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:44.457737 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:44.459416 (+ 1679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.459958 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.460649 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.460854 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.463129 (+ 2275us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:44.463151 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:44.463154 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:44.463187 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:44.464774 (+ 1587us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:44.465271 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:44.465273 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:44.465275 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:44.465323 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:44.465577 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:44.465580 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:44.465581 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:44.465763 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:44.465837 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:44.466351 (+ 514us) server_negotiation.cc:300] Negotiation successful
0201 03:03:44.466476 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260201 03:03:44.467308 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.456216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:39937 (local address 127.18.203.195:57889)
0201 03:03:44.456616 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:44.456651 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:44.457400 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:44.457906 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:44.457914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:44.458317 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:03:44.459264 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.459278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.460081 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.460084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:44.460522 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.460531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.460757 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.461408 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:03:44.461435 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:03:44.462951 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:03:44.464891 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:44.464898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:44.464912 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:44.465178 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:44.465399 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:03:44.465402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:03:44.465403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:03:44.465510 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:03:44.465837 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:03:44.465843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:03:44.466120 (+ 277us) client_negotiation.cc:769] Sending connection context
0201 03:03:44.466319 (+ 199us) client_negotiation.cc:240] Negotiation successful
0201 03:03:44.466562 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
I20260201 03:03:44.468359 30642 heartbeater.cc:344] Connected to a master server at 127.18.203.254:39937
I20260201 03:03:44.468598 30642 heartbeater.cc:461] Registering TS with master...
I20260201 03:03:44.469102 30642 heartbeater.cc:507] Master 127.18.203.254:39937 requested a full tablet report, sending...
I20260201 03:03:44.470049 30177 ts_manager.cc:194] Registered new tserver with Master: 4ac9809d418b466cb0237fe09b74f147 (127.18.203.195:35491)
I20260201 03:03:44.470607 30177 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:57889
I20260201 03:03:44.476562 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260201 03:03:44.484627 30216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.478423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.18.203.254:39937)
0201 03:03:44.478564 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:44.478569 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:44.478687 (+ 118us) server_negotiation.cc:408] Connection header received
0201 03:03:44.478816 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:44.478819 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:44.478863 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:44.478923 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:03:44.479785 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.480282 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.480919 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.481098 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.482069 (+ 971us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:03:44.482087 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:03:44.482089 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:03:44.482113 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:03:44.483215 (+ 1102us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:44.483575 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:44.483578 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:44.483579 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:44.483656 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:03:44.483857 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:03:44.483861 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:03:44.483863 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:03:44.484036 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:03:44.484158 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:44.484359 (+ 201us) server_negotiation.cc:300] Negotiation successful
0201 03:03:44.484469 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260201 03:03:44.487164 30177 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:03:44.489557 30177 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:03:44.502882 30653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.498022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:37405 (local address 127.0.0.1:49202)
0201 03:03:44.498538 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:44.498579 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:44.498725 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:44.499922 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:44.499925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:44.499944 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:44.500178 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.500186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.501520 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.501524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:44.502424 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.502435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.502527 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.502557 (+ 30us) client_negotiation.cc:769] Sending connection context
0201 03:03:44.502609 (+ 52us) client_negotiation.cc:240] Negotiation successful
0201 03:03:44.502695 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":115,"threads_started":1}
I20260201 03:03:44.502882 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.498799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:33337 (local address 127.0.0.1:50998)
0201 03:03:44.499947 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:44.499960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:44.500040 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:44.500462 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:44.500465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:44.500483 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:44.500668 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.500674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.501649 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.501653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:44.502454 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.502461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.502546 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.502558 (+ 12us) client_negotiation.cc:769] Sending connection context
0201 03:03:44.502621 (+ 63us) client_negotiation.cc:240] Negotiation successful
0201 03:03:44.502701 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"thread_start_us":107,"threads_started":1}
I20260201 03:03:44.503820 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.498188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:35491 (local address 127.0.0.1:55848)
0201 03:03:44.499066 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:44.499080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:44.499180 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:44.499705 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:44.499710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:44.499734 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:44.500033 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.500040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.501424 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.501428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:44.502336 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.502345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.502448 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.502528 (+ 80us) client_negotiation.cc:769] Sending connection context
0201 03:03:44.502622 (+ 94us) client_negotiation.cc:240] Negotiation successful
0201 03:03:44.502702 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":114,"threads_started":1}
I20260201 03:03:44.504135 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.498618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.18.203.195:35491)
0201 03:03:44.499194 (+ 576us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:44.499202 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:44.499228 (+ 26us) server_negotiation.cc:408] Connection header received
0201 03:03:44.499309 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:44.499313 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:44.499505 (+ 192us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:44.499675 (+ 170us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:44.500157 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.501307 (+ 1150us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.502792 (+ 1485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.503364 (+ 572us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.503513 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:44.503584 (+ 71us) server_negotiation.cc:300] Negotiation successful
0201 03:03:44.503999 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":148,"threads_started":1}
I20260201 03:03:44.504186 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.499096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.18.203.194:33337)
0201 03:03:44.499562 (+ 466us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:44.499568 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:44.500083 (+ 515us) server_negotiation.cc:408] Connection header received
0201 03:03:44.500165 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:44.500170 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:44.500341 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:44.500458 (+ 117us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:44.500780 (+ 322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.501529 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.502793 (+ 1264us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.503648 (+ 855us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.503882 (+ 234us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:44.503951 (+ 69us) server_negotiation.cc:300] Negotiation successful
0201 03:03:44.504050 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":116,"threads_started":1}
I20260201 03:03:44.504482 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.498122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.18.203.193:37405)
0201 03:03:44.499501 (+ 1379us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:44.499507 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:44.499530 (+ 23us) server_negotiation.cc:408] Connection header received
0201 03:03:44.499654 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:44.499660 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:44.499794 (+ 134us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:44.499924 (+ 130us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:44.500304 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.501371 (+ 1067us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.503282 (+ 1911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.504071 (+ 789us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.504191 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:44.504259 (+ 68us) server_negotiation.cc:300] Negotiation successful
0201 03:03:44.504349 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1271,"thread_start_us":64,"threads_started":1}
I20260201 03:03:44.506826 30304 tablet_service.cc:1505] Processing CreateTablet for tablet 486461ce5cbc475b808b76736d8855eb (DEFAULT_TABLE table=test-table [id=7d54dd684e5048db855fae0b6b576344]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:44.506939 30440 tablet_service.cc:1505] Processing CreateTablet for tablet 486461ce5cbc475b808b76736d8855eb (DEFAULT_TABLE table=test-table [id=7d54dd684e5048db855fae0b6b576344]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:44.506939 30576 tablet_service.cc:1505] Processing CreateTablet for tablet 486461ce5cbc475b808b76736d8855eb (DEFAULT_TABLE table=test-table [id=7d54dd684e5048db855fae0b6b576344]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:03:44.507748 30304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486461ce5cbc475b808b76736d8855eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:44.507825 30440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486461ce5cbc475b808b76736d8855eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:44.507825 30576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486461ce5cbc475b808b76736d8855eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:03:44.513089 30659 tablet_bootstrap.cc:492] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2: Bootstrap starting.
I20260201 03:03:44.514642 30660 tablet_bootstrap.cc:492] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d: Bootstrap starting.
I20260201 03:03:44.514642 30661 tablet_bootstrap.cc:492] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147: Bootstrap starting.
I20260201 03:03:44.515939 30659 tablet_bootstrap.cc:654] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:44.516646 30659 log.cc:826] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:44.516770 30661 tablet_bootstrap.cc:654] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:44.516769 30660 tablet_bootstrap.cc:654] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d: Neither blocks nor log segments found. Creating new log.
I20260201 03:03:44.517591 30660 log.cc:826] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:44.517591 30661 log.cc:826] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147: Log is configured to *not* fsync() on all Append() calls
I20260201 03:03:44.518468 30659 tablet_bootstrap.cc:492] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2: No bootstrap required, opened a new log
I20260201 03:03:44.518671 30659 ts_tablet_manager.cc:1403] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260201 03:03:44.519243 30660 tablet_bootstrap.cc:492] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d: No bootstrap required, opened a new log
I20260201 03:03:44.519382 30661 tablet_bootstrap.cc:492] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147: No bootstrap required, opened a new log
I20260201 03:03:44.519418 30660 ts_tablet_manager.cc:1403] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260201 03:03:44.519542 30661 ts_tablet_manager.cc:1403] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260201 03:03:44.521906 30659 raft_consensus.cc:359] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d472685386498ea2c5e7dbd8672de2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 37405 } } peers { permanent_uuid: "4ac9809d418b466cb0237fe09b74f147" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 35491 } } peers { permanent_uuid: "c8364859d2044a7896189d8a0c502d0d" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33337 } }
I20260201 03:03:44.522163 30659 raft_consensus.cc:385] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:44.522220 30659 raft_consensus.cc:740] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d472685386498ea2c5e7dbd8672de2, State: Initialized, Role: FOLLOWER
I20260201 03:03:44.522270 30660 raft_consensus.cc:359] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d472685386498ea2c5e7dbd8672de2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 37405 } } peers { permanent_uuid: "4ac9809d418b466cb0237fe09b74f147" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 35491 } } peers { permanent_uuid: "c8364859d2044a7896189d8a0c502d0d" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33337 } }
I20260201 03:03:44.522473 30660 raft_consensus.cc:385] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:44.522540 30660 raft_consensus.cc:740] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8364859d2044a7896189d8a0c502d0d, State: Initialized, Role: FOLLOWER
I20260201 03:03:44.522686 30659 consensus_queue.cc:260] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d472685386498ea2c5e7dbd8672de2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 37405 } } peers { permanent_uuid: "4ac9809d418b466cb0237fe09b74f147" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 35491 } } peers { permanent_uuid: "c8364859d2044a7896189d8a0c502d0d" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33337 } }
I20260201 03:03:44.522929 30660 consensus_queue.cc:260] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d472685386498ea2c5e7dbd8672de2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 37405 } } peers { permanent_uuid: "4ac9809d418b466cb0237fe09b74f147" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 35491 } } peers { permanent_uuid: "c8364859d2044a7896189d8a0c502d0d" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33337 } }
I20260201 03:03:44.522969 30661 raft_consensus.cc:359] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d472685386498ea2c5e7dbd8672de2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 37405 } } peers { permanent_uuid: "4ac9809d418b466cb0237fe09b74f147" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 35491 } } peers { permanent_uuid: "c8364859d2044a7896189d8a0c502d0d" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33337 } }
I20260201 03:03:44.523207 30661 raft_consensus.cc:385] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:03:44.523262 30661 raft_consensus.cc:740] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac9809d418b466cb0237fe09b74f147, State: Initialized, Role: FOLLOWER
I20260201 03:03:44.523687 30659 ts_tablet_manager.cc:1434] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260201 03:03:44.523694 30370 heartbeater.cc:499] Master 127.18.203.254:39937 was elected leader, sending a full tablet report...
I20260201 03:03:44.523687 30661 consensus_queue.cc:260] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d472685386498ea2c5e7dbd8672de2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 37405 } } peers { permanent_uuid: "4ac9809d418b466cb0237fe09b74f147" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 35491 } } peers { permanent_uuid: "c8364859d2044a7896189d8a0c502d0d" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33337 } }
I20260201 03:03:44.523785 30660 ts_tablet_manager.cc:1434] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260201 03:03:44.524097 30506 heartbeater.cc:499] Master 127.18.203.254:39937 was elected leader, sending a full tablet report...
I20260201 03:03:44.524375 30661 ts_tablet_manager.cc:1434] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260201 03:03:44.524673 30642 heartbeater.cc:499] Master 127.18.203.254:39937 was elected leader, sending a full tablet report...
I20260201 03:03:44.538968 30667 raft_consensus.cc:493] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:03:44.539171 30667 raft_consensus.cc:515] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d472685386498ea2c5e7dbd8672de2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 37405 } } peers { permanent_uuid: "4ac9809d418b466cb0237fe09b74f147" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 35491 } } peers { permanent_uuid: "c8364859d2044a7896189d8a0c502d0d" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33337 } }
I20260201 03:03:44.540401 30667 leader_election.cc:290] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34d472685386498ea2c5e7dbd8672de2 (127.18.203.193:37405), c8364859d2044a7896189d8a0c502d0d (127.18.203.194:33337)
I20260201 03:03:44.543861 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.540639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:37405 (local address 127.18.203.195:41723)
0201 03:03:44.540836 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:44.540852 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:44.540950 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:44.541183 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:44.541187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:44.541212 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:44.541454 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.541462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.542426 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.542429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:44.543388 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.543398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.543514 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.543538 (+ 24us) client_negotiation.cc:769] Sending connection context
0201 03:03:44.543589 (+ 51us) client_negotiation.cc:240] Negotiation successful
0201 03:03:44.543676 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"spinlock_wait_cycles":2304}
I20260201 03:03:44.543874 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.540643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:33337 (local address 127.18.203.195:59081)
0201 03:03:44.541108 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:44.541120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0201 03:03:44.541206 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:03:44.541407 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:03:44.541410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:03:44.541425 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:03:44.541656 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.541662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.542520 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.542523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:03:44.543393 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:03:44.543399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.543514 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.543538 (+ 24us) client_negotiation.cc:769] Sending connection context
0201 03:03:44.543589 (+ 51us) client_negotiation.cc:240] Negotiation successful
0201 03:03:44.543676 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":47,"thread_start_us":142,"threads_started":1}
I20260201 03:03:44.544301 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.540828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:59081 (local address 127.18.203.194:33337)
0201 03:03:44.540961 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:44.540966 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:44.541194 (+ 228us) server_negotiation.cc:408] Connection header received
0201 03:03:44.541280 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:44.541282 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:44.541323 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:44.541382 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:44.541750 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.542434 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.543509 (+ 1075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.544032 (+ 523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.544072 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:44.544122 (+ 50us) server_negotiation.cc:300] Negotiation successful
0201 03:03:44.544184 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260201 03:03:44.544301 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.540843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:41723 (local address 127.18.203.193:37405)
0201 03:03:44.540961 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:44.540966 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:44.540981 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:03:44.541032 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:44.541035 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:44.541077 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:44.541145 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:03:44.541573 (+ 428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.542305 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.543515 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.544033 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.544072 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:44.544121 (+ 49us) server_negotiation.cc:300] Negotiation successful
0201 03:03:44.544182 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260201 03:03:44.544900 30324 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "486461ce5cbc475b808b76736d8855eb" candidate_uuid: "4ac9809d418b466cb0237fe09b74f147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d472685386498ea2c5e7dbd8672de2" is_pre_election: true
I20260201 03:03:44.544901 30460 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "486461ce5cbc475b808b76736d8855eb" candidate_uuid: "4ac9809d418b466cb0237fe09b74f147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8364859d2044a7896189d8a0c502d0d" is_pre_election: true
I20260201 03:03:44.545161 30324 raft_consensus.cc:2468] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ac9809d418b466cb0237fe09b74f147 in term 0.
I20260201 03:03:44.545195 30460 raft_consensus.cc:2468] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ac9809d418b466cb0237fe09b74f147 in term 0.
I20260201 03:03:44.545600 30531 leader_election.cc:304] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d472685386498ea2c5e7dbd8672de2, 4ac9809d418b466cb0237fe09b74f147; no voters:
I20260201 03:03:44.545884 30667 raft_consensus.cc:2804] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:03:44.545969 30667 raft_consensus.cc:493] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:03:44.546011 30667 raft_consensus.cc:3060] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:44.547158 30667 raft_consensus.cc:515] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d472685386498ea2c5e7dbd8672de2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 37405 } } peers { permanent_uuid: "4ac9809d418b466cb0237fe09b74f147" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 35491 } } peers { permanent_uuid: "c8364859d2044a7896189d8a0c502d0d" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33337 } }
I20260201 03:03:44.547511 30667 leader_election.cc:290] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [CANDIDATE]: Term 1 election: Requested vote from peers 34d472685386498ea2c5e7dbd8672de2 (127.18.203.193:37405), c8364859d2044a7896189d8a0c502d0d (127.18.203.194:33337)
I20260201 03:03:44.547874 30460 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "486461ce5cbc475b808b76736d8855eb" candidate_uuid: "4ac9809d418b466cb0237fe09b74f147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8364859d2044a7896189d8a0c502d0d"
I20260201 03:03:44.547878 30324 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "486461ce5cbc475b808b76736d8855eb" candidate_uuid: "4ac9809d418b466cb0237fe09b74f147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d472685386498ea2c5e7dbd8672de2"
I20260201 03:03:44.547991 30324 raft_consensus.cc:3060] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:44.547986 30460 raft_consensus.cc:3060] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:03:44.548985 30460 raft_consensus.cc:2468] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ac9809d418b466cb0237fe09b74f147 in term 1.
I20260201 03:03:44.549350 30529 leader_election.cc:304] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac9809d418b466cb0237fe09b74f147, c8364859d2044a7896189d8a0c502d0d; no voters:
I20260201 03:03:44.549396 30324 raft_consensus.cc:2468] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ac9809d418b466cb0237fe09b74f147 in term 1.
I20260201 03:03:44.549543 30667 raft_consensus.cc:2804] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:03:44.549813 30667 raft_consensus.cc:697] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 1 LEADER]: Becoming Leader. State: Replica: 4ac9809d418b466cb0237fe09b74f147, State: Running, Role: LEADER
I20260201 03:03:44.550181 30667 consensus_queue.cc:237] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d472685386498ea2c5e7dbd8672de2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 37405 } } peers { permanent_uuid: "4ac9809d418b466cb0237fe09b74f147" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 35491 } } peers { permanent_uuid: "c8364859d2044a7896189d8a0c502d0d" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33337 } }
I20260201 03:03:44.553687 30177 catalog_manager.cc:5654] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ac9809d418b466cb0237fe09b74f147 (127.18.203.195). New cstate: current_term: 1 leader_uuid: "4ac9809d418b466cb0237fe09b74f147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d472685386498ea2c5e7dbd8672de2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ac9809d418b466cb0237fe09b74f147" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8364859d2044a7896189d8a0c502d0d" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33337 } health_report { overall_health: UNKNOWN } } }
I20260201 03:03:44.567035 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.563927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.18.203.195:35491)
0201 03:03:44.564071 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0201 03:03:44.564076 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:03:44.564092 (+ 16us) server_negotiation.cc:408] Connection header received
0201 03:03:44.564207 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:03:44.564210 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:03:44.564268 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:03:44.564356 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:03:44.564726 (+ 370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.565235 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:03:44.565892 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:03:44.566075 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:03:44.566185 (+ 110us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:03:44.566647 (+ 462us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:03:44.566773 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0201 03:03:44.566843 (+ 70us) server_negotiation.cc:300] Negotiation successful
0201 03:03:44.566896 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260201 03:03:44.572676 30324 raft_consensus.cc:1275] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2 [term 1 FOLLOWER]: Refusing update from remote peer 4ac9809d418b466cb0237fe09b74f147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:44.572676 30460 raft_consensus.cc:1275] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d [term 1 FOLLOWER]: Refusing update from remote peer 4ac9809d418b466cb0237fe09b74f147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:03:44.573355 30669 consensus_queue.cc:1048] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8364859d2044a7896189d8a0c502d0d" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 33337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:44.573570 30667 consensus_queue.cc:1048] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34d472685386498ea2c5e7dbd8672de2" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:03:44.582710 30674 mvcc.cc:204] Tried to move back new op lower bound from 7249571940642299904 to 7249571940560617472. Current Snapshot: MvccSnapshot[applied={T|T < 7249571940642299904 or (T in {7249571940642299904})}]
I20260201 03:03:44.583067 30673 mvcc.cc:204] Tried to move back new op lower bound from 7249571940642299904 to 7249571940560617472. Current Snapshot: MvccSnapshot[applied={T|T < 7249571940642299904}]
I20260201 03:03:44.589154 30174 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:44.589336 30174 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:03:44.591958 30174 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c4731cabcf764559b259013bba5733cf: Sending DeleteTablet for 3 replicas of tablet 486461ce5cbc475b808b76736d8855eb
I20260201 03:03:44.592747 30576 tablet_service.cc:1552] Processing DeleteTablet for tablet 486461ce5cbc475b808b76736d8855eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:44 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:55848
I20260201 03:03:44.592825 30440 tablet_service.cc:1552] Processing DeleteTablet for tablet 486461ce5cbc475b808b76736d8855eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:44 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:50998
I20260201 03:03:44.592803 30304 tablet_service.cc:1552] Processing DeleteTablet for tablet 486461ce5cbc475b808b76736d8855eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:03:44 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:49202
I20260201 03:03:44.593346 30683 tablet_replica.cc:333] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2: stopping tablet replica
I20260201 03:03:44.593644 30683 raft_consensus.cc:2243] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:44.593698 30681 tablet_replica.cc:333] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d: stopping tablet replica
I20260201 03:03:44.593909 30681 raft_consensus.cc:2243] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:03:44.594000 30683 raft_consensus.cc:2272] T 486461ce5cbc475b808b76736d8855eb P 34d472685386498ea2c5e7dbd8672de2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:44.594167 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 30241
I20260201 03:03:44.594175 30681 raft_consensus.cc:2272] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:44.595221 30682 tablet_replica.cc:333] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147: stopping tablet replica
I20260201 03:03:44.595440 30682 raft_consensus.cc:2243] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:03:44.595882 30682 raft_consensus.cc:2272] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:03:44.595906 30681 ts_tablet_manager.cc:1916] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:44.598733 30681 ts_tablet_manager.cc:1929] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:44.598826 30681 log.cc:1199] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/486461ce5cbc475b808b76736d8855eb
I20260201 03:03:44.599159 30681 ts_tablet_manager.cc:1950] T 486461ce5cbc475b808b76736d8855eb P c8364859d2044a7896189d8a0c502d0d: Deleting consensus metadata
I20260201 03:03:44.599256 30682 ts_tablet_manager.cc:1916] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:03:44.600392 30161 catalog_manager.cc:4985] TS c8364859d2044a7896189d8a0c502d0d (127.18.203.194:33337): tablet 486461ce5cbc475b808b76736d8855eb (table test-table [id=7d54dd684e5048db855fae0b6b576344]) successfully deleted
I20260201 03:03:44.601948 30682 ts_tablet_manager.cc:1929] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:03:44.602058 30682 log.cc:1199] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/486461ce5cbc475b808b76736d8855eb
I20260201 03:03:44.602413 30682 ts_tablet_manager.cc:1950] T 486461ce5cbc475b808b76736d8855eb P 4ac9809d418b466cb0237fe09b74f147: Deleting consensus metadata
W20260201 03:03:44.602967 30163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.203.193:37405 (error 108)
I20260201 03:03:44.603659 30162 catalog_manager.cc:4985] TS 4ac9809d418b466cb0237fe09b74f147 (127.18.203.195:35491): tablet 486461ce5cbc475b808b76736d8855eb (table test-table [id=7d54dd684e5048db855fae0b6b576344]) successfully deleted
I20260201 03:03:44.604017 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:03:44.603715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:37405 (local address 127.0.0.1:49210)
0201 03:03:44.603887 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0201 03:03:44.603943 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.193:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260201 03:03:44.604157 30163 catalog_manager.cc:4712] TS 34d472685386498ea2c5e7dbd8672de2 (127.18.203.193:37405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 486461ce5cbc475b808b76736d8855eb: Network error: Client connection negotiation failed: client connection to 127.18.203.193:37405: connect: Connection refused (error 111)
I20260201 03:03:44.604458 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 30377
I20260201 03:03:44.611182 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 30513
I20260201 03:03:44.617373 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 30145
2026-02-01T03:03:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3083 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:03:44 dist-test-slave-gf59 krb5kdc[30687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:03:44 dist-test-slave-gf59 krb5kdc[30687](info): set up 2 sockets
Feb 01 03:03:44 dist-test-slave-gf59 krb5kdc[30687](info): commencing operation
krb5kdc: starting...
W20260201 03:03:46.464432 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.825s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:03:46 dist-test-slave-gf59 krb5kdc[30687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:03:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:03:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/rangeradmin_127.18.203.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/rangeradmin_127.18.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/rangerlookup_127.18.203.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/rangerlookup_127.18.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/HTTP_127.18.203.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/HTTP_127.18.203.212@KRBTEST.COM.keytab.
I20260201 03:03:46.631181 19247 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyeWP68/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/postgres -l logfile start
2026-02-01 03:03:48.987 UTC [30718] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-02-01 03:03:48.987 UTC [30718] LOG: listening on IPv4 address "127.18.203.212", port 42981
2026-02-01 03:03:48.991 UTC [30718] LOG: listening on Unix socket "/tmp/.s.PGSQL.42981"
2026-02-01 03:03:48.996 UTC [30723] LOG: database system was shut down at 2026-02-01 03:03:47 UTC
2026-02-01 03:03:49.001 UTC [30718] LOG: database system is ready to accept connections
I20260201 03:03:50.756067 19247 mini_postgres.cc:96] Postgres bound to 42981
2026-02-01 03:03:50.762 UTC [30729] FATAL: database "slave" does not exist
127.18.203.212:42981 - accepting connections
I20260201 03:03:50.763352 19247 mini_ranger.cc:161] Starting Ranger...
I20260201 03:03:50.781541 19247 mini_ranger.cc:84] Created miniranger Postgres user
I20260201 03:03:50.847702 19247 mini_ranger.cc:87] Created ranger Postgres database
I20260201 03:03:50.847811 19247 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-admin
2026-02-01 03:03:51,122 [I] DB FLAVOR :POSTGRES
2026-02-01 03:03:51,122 [I] --------- Verifying Ranger DB connection ---------
2026-02-01 03:03:51,122 [I] Checking connection..
2026-02-01 03:03:51,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-02-01 03:03:51,424 [I] Checking connection passed.
2026-02-01 03:03:51,424 [I] --------- Verifying version history table ---------
2026-02-01 03:03:51,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-01 03:03:51,720 [I] Table x_db_version_h does not exist in database ranger
2026-02-01 03:03:51,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-01 03:03:52,013 [I] Table x_db_version_h does not exist in database ranger
2026-02-01 03:03:52,014 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-02-01 03:03:52,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-02-01 03:03:52.298 UTC [30796] WARNING: there is no transaction in progress
2026-02-01 03:03:52,305 [I] create_dbversion_catalog.sql file imported successfully
2026-02-01 03:03:52,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-01 03:03:52,597 [I] Table x_db_version_h already exists in database 'ranger'
2026-02-01 03:03:52,598 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-01 03:03:52,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-02-01 03:03:52,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-02-01 03:03:53,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gf59.c.gcp-upstream.internal','N') ;"
2026-02-01 03:03:53,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-01 03:03:53,810 [I] Table x_portal_user does not exist in database ranger
2026-02-01 03:03:53,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-01 03:03:54,093 [I] Table x_policy_ref_group does not exist in database ranger
2026-02-01 03:03:54,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-01 03:03:54,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-01 03:03:54,675 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-02-01 03:03:54,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-02-01 03:03:55.463 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.474 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.485 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.495 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.505 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.515 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.558 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.563 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.572 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.584 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.595 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.604 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.611 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.615 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.618 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55.621 UTC [30913] WARNING: there is no transaction in progress
2026-02-01 03:03:55,980 [I] ranger_core_db_postgres.sql file imported successfully
2026-02-01 03:03:55,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-02-01 03:03:56,266 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-02-01 03:03:56,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-01 03:03:56,564 [I] Table x_portal_user already exists in database 'ranger'
2026-02-01 03:03:56,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-01 03:03:56,886 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-02-01 03:03:56,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-01 03:03:57,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-01 03:03:57,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gf59.c.gcp-upstream.internal';"
2026-02-01 03:03:57,762 [I] CORE_DB_SCHEMA import status has been updated
2026-02-01 03:03:57,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-02-01 03:03:58,052 [I] DB_PATCHES have already been applied
I20260201 03:03:58.058364 19247 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyeWP68/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260201 03:03:58.058436 19247 mini_ranger.cc:193] Using host: 127.18.203.212
I20260201 03:03:58.062216 19247 mini_ranger.cc:227] Ranger admin URL: http://127.18.203.212:35949
Feb 01, 2026 3:03:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 01, 2026 3:03:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 01, 2026 3:03:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-admin/ews/webapp
Feb 01, 2026 3:03:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Feb 01, 2026 3:03:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-admin/ews/webapp] .....
Feb 01, 2026 3:03:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-admin/ews/webapp].
Feb 01, 2026 3:03:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260201 03:04:00.088690 19247 mini_ranger.cc:160] Time spent starting Ranger: real 9.325s user 0.008s sys 0.004s
Feb 01, 2026 3:04:18 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Feb 01, 2026 3:04:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.UserREST
Feb 01, 2026 3:04:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Feb 01, 2026 3:04:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Feb 01, 2026 3:04:18 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Feb 01, 2026 3:04:19 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260201 03:04:19.927814 19247 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/rangerkms_127.18.203.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/rangerkms_127.18.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.203.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.203.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/HTTP_127.18.203.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.203.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/HTTP_127.18.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Feb 01 03:04:19 dist-test-slave-gf59 krb5kdc[30687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915059, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260201 03:04:20.006676 19247 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260201 03:04:20.017026 19247 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260201 03:04:20.119764 19247 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260201 03:04:20.119884 19247 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-kms
I20260201 03:04:20.119925 19247 mini_ranger_kms.cc:227] Using postgres at 127.18.203.212:42981
2026-02-01 03:04:20,287 [I] DB FLAVOR :POSTGRES
2026-02-01 03:04:20,287 [I] --------- Verifying Ranger DB connection ---------
2026-02-01 03:04:20,287 [I] Checking connection
2026-02-01 03:04:20,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-02-01 03:04:20,595 [I] connection success
2026-02-01 03:04:20,595 [I] --------- Verifying Ranger DB tables ---------
2026-02-01 03:04:20,595 [I] Verifying table ranger_masterkey in database rangerkms
2026-02-01 03:04:20,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-02-01 03:04:20,906 [I] Table ranger_masterkey does not exist in database rangerkms
2026-02-01 03:04:20,906 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-01 03:04:20,906 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-02-01 03:04:20,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:42981/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-02-01 03:04:21,240 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260201 03:04:21.686007 19247 mini_ranger_kms.cc:310] Created kms service
I20260201 03:04:21.903366 19247 mini_ranger_kms.cc:326] Created kudu user
I20260201 03:04:21.948019 19247 mini_ranger_kms.cc:343] Created rangerkms user
Feb 01 03:04:21 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915059, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
I20260201 03:04:22.283553 19247 mini_ranger_kms.cc:384] Added ranger policy
I20260201 03:04:22.283807 19247 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260201 03:04:22.283833 19247 mini_ranger_kms.cc:240] Using host: 127.18.203.212
I20260201 03:04:22.287799 19247 mini_ranger_kms.cc:276] Ranger KMS PID: 31105
I20260201 03:04:22.287916 19247 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.203.212:57641
Feb 01, 2026 3:04:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 01, 2026 3:04:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 01, 2026 3:04:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Feb 01, 2026 3:04:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-kms/ews/webapp] .....
Feb 01, 2026 3:04:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-kms/ews/webapp].
Feb 01, 2026 3:04:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260201 03:04:24.250907 19247 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.244s user 0.021s sys 0.131s
I20260201 03:04:24.251036 19247 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260201 03:04:24.251127 19247 mini_ranger_kms.cc:401] 127.18.203.212:57641/kms/v1/keys
Feb 01 03:04:24 dist-test-slave-gf59 krb5kdc[30687](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1769915064, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.203.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Feb 01 03:04:24 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915064, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.203.212@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
Feb 01 03:04:25 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915064, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.203.212@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
Feb 01 03:04:27 dist-test-slave-gf59 krb5kdc[30687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:04:27.378296 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:37471
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:37471
--ranger_config_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.203.212:57641/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:04:27.486758 31158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:04:27.487020 31158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:04:27.487092 31158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:04:27.490355 31158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:04:27.490430 31158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:04:27.490463 31158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:04:27.490487 31158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:04:27.490522 31158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:04:27.494920 31158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.203.212:57641/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:37471
--ranger_config_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:37471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:04:27.496213 31158 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:04:27.497098 31158 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260201 03:04:27.503126 31158 server_base.cc:1060] running on GCE node
W20260201 03:04:27.503131 31163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:04:27.503057 31164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:04:27.503058 31166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:04:27.503999 31158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:04:27.505129 31158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:04:27.506317 31158 hybrid_clock.cc:648] HybridClock initialized: now 1769915067506298 us; error 32 us; skew 500 ppm
Feb 01 03:04:27 dist-test-slave-gf59 krb5kdc[30687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915067, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:04:27.509814 31158 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:04:27.511184 31158 webserver.cc:492] Webserver started at http://127.18.203.254:42937/ using document root <none> and password file <none>
I20260201 03:04:27.511910 31158 fs_manager.cc:362] Metadata directory not provided
I20260201 03:04:27.511986 31158 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:04:27.512216 31158 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 01 03:04:27 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915067, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
I20260201 03:04:27.556702 31158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "bead5d100ab647c9adc567a921aaf355"
format_stamp: "Formatted at 2026-02-01 03:04:27 on dist-test-slave-gf59"
server_key: "e4f92ec6f62ba881014a910feed9b946"
server_key_iv: "13e924b056e4c7419dd4379c22d28706"
server_key_version: "kuduclusterkey@0"
I20260201 03:04:27.557236 31158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "bead5d100ab647c9adc567a921aaf355"
format_stamp: "Formatted at 2026-02-01 03:04:27 on dist-test-slave-gf59"
server_key: "e4f92ec6f62ba881014a910feed9b946"
server_key_iv: "13e924b056e4c7419dd4379c22d28706"
server_key_version: "kuduclusterkey@0"
I20260201 03:04:27.560662 31158 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260201 03:04:27.577711 31173 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:04:27.578910 31158 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260201 03:04:27.579032 31158 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "bead5d100ab647c9adc567a921aaf355"
format_stamp: "Formatted at 2026-02-01 03:04:27 on dist-test-slave-gf59"
server_key: "e4f92ec6f62ba881014a910feed9b946"
server_key_iv: "13e924b056e4c7419dd4379c22d28706"
server_key_version: "kuduclusterkey@0"
I20260201 03:04:27.579146 31158 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:04:27.593792 31158 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:04:27.596689 31158 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:04:27.596873 31158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:04:27.604692 31158 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:37471
I20260201 03:04:27.604701 31225 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:37471 every 8 connection(s)
I20260201 03:04:27.605783 31158 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:04:27.609197 31226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:04:27.615520 31226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355: Bootstrap starting.
I20260201 03:04:27.615542 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 31158
I20260201 03:04:27.615698 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:04:27.615950 19247 external_mini_cluster.cc:1468] Setting key ced304ecdc0182ab2b60bb25c4f3936c
I20260201 03:04:27.618186 31226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355: Neither blocks nor log segments found. Creating new log.
I20260201 03:04:27.619068 31226 log.cc:826] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355: Log is configured to *not* fsync() on all Append() calls
I20260201 03:04:27.621021 31226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355: No bootstrap required, opened a new log
Feb 01 03:04:27 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:04:27.624262 31226 raft_consensus.cc:359] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead5d100ab647c9adc567a921aaf355" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 37471 } }
I20260201 03:04:27.624473 31226 raft_consensus.cc:385] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:04:27.624569 31226 raft_consensus.cc:740] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bead5d100ab647c9adc567a921aaf355, State: Initialized, Role: FOLLOWER
I20260201 03:04:27.625066 31226 consensus_queue.cc:260] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead5d100ab647c9adc567a921aaf355" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 37471 } }
I20260201 03:04:27.625196 31226 raft_consensus.cc:399] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:04:27.625273 31226 raft_consensus.cc:493] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:04:27.625380 31226 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:04:27.626273 31226 raft_consensus.cc:515] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead5d100ab647c9adc567a921aaf355" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 37471 } }
I20260201 03:04:27.626612 31226 leader_election.cc:304] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bead5d100ab647c9adc567a921aaf355; no voters:
I20260201 03:04:27.626926 31226 leader_election.cc:290] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:04:27.627014 31231 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:04:27.627270 31231 raft_consensus.cc:697] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [term 1 LEADER]: Becoming Leader. State: Replica: bead5d100ab647c9adc567a921aaf355, State: Running, Role: LEADER
I20260201 03:04:27.627637 31231 consensus_queue.cc:237] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead5d100ab647c9adc567a921aaf355" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 37471 } }
I20260201 03:04:27.628242 31226 sys_catalog.cc:565] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:04:27.628851 31231 sys_catalog.cc:455] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bead5d100ab647c9adc567a921aaf355. Latest consensus state: current_term: 1 leader_uuid: "bead5d100ab647c9adc567a921aaf355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead5d100ab647c9adc567a921aaf355" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 37471 } } }
I20260201 03:04:27.629043 31231 sys_catalog.cc:458] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [sys.catalog]: This master's current role is: LEADER
I20260201 03:04:27.629287 31231 sys_catalog.cc:455] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bead5d100ab647c9adc567a921aaf355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead5d100ab647c9adc567a921aaf355" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 37471 } } }
I20260201 03:04:27.629419 31231 sys_catalog.cc:458] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355 [sys.catalog]: This master's current role is: LEADER
I20260201 03:04:27.629730 31238 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:04:27.630535 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:27.617206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.18.203.254:37471)
0201 03:04:27.617607 (+ 401us) server_negotiation.cc:207] Beginning negotiation
0201 03:04:27.617616 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0201 03:04:27.617652 (+ 36us) server_negotiation.cc:408] Connection header received
0201 03:04:27.618251 (+ 599us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:04:27.618279 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:04:27.618610 (+ 331us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:04:27.618924 (+ 314us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:04:27.619953 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:27.621110 (+ 1157us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:04:27.622034 (+ 924us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:27.622314 (+ 280us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:27.625009 (+ 2695us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:04:27.625031 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:04:27.625045 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:04:27.625087 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:04:27.627004 (+ 1917us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:04:27.627460 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:04:27.627466 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:04:27.627471 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:04:27.627549 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:04:27.627870 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:04:27.627873 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:04:27.627875 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:04:27.628338 (+ 463us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:04:27.629382 (+ 1044us) server_negotiation.cc:1036] Waiting for connection context
0201 03:04:27.629548 (+ 166us) server_negotiation.cc:300] Negotiation successful
0201 03:04:27.629782 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260201 03:04:27.633422 31238 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:04:27.633872 31226 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260201 03:04:27.642889 31238 catalog_manager.cc:1357] Generated new cluster ID: c9d621dd4d0b4641b9fc667f91acd7c7
I20260201 03:04:27.643050 31238 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:04:27.660705 31238 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:04:27.662192 31238 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:04:27.674774 31238 catalog_manager.cc:6027] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355: Generated new TSK 0
I20260201 03:04:27.677007 31238 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:04:28.443567 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.203.212:57641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:37471
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:04:28.579291 31272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:04:28.579663 31272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:04:28.579815 31272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:04:28.584647 31272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:04:28.584750 31272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:04:28.584869 31272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:04:28.591105 31272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.203.212:57641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:37471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:04:28.592726 31272 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:04:28.593914 31272 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:04:28.602095 31279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:04:28.602301 31280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:04:28.602409 31272 server_base.cc:1060] running on GCE node
W20260201 03:04:28.602519 31282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:04:28.602854 31272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:04:28.603410 31272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:04:28.604562 31272 hybrid_clock.cc:648] HybridClock initialized: now 1769915068604529 us; error 56 us; skew 500 ppm
Feb 01 03:04:28 dist-test-slave-gf59 krb5kdc[30687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915068, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:04:28.607066 31272 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:04:28.608172 31272 webserver.cc:492] Webserver started at http://127.18.203.193:46843/ using document root <none> and password file <none>
I20260201 03:04:28.608711 31272 fs_manager.cc:362] Metadata directory not provided
I20260201 03:04:28.608762 31272 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:04:28.608932 31272 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 01 03:04:28 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915068, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
I20260201 03:04:28.627693 31272 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "9d831bbc48344df498a0a939cb6e2792"
format_stamp: "Formatted at 2026-02-01 03:04:28 on dist-test-slave-gf59"
server_key: "d63500b16022654a6ac8f6161214bfda"
server_key_iv: "f62c51009db84c30977408603028de53"
server_key_version: "kuduclusterkey@0"
I20260201 03:04:28.628389 31272 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "9d831bbc48344df498a0a939cb6e2792"
format_stamp: "Formatted at 2026-02-01 03:04:28 on dist-test-slave-gf59"
server_key: "d63500b16022654a6ac8f6161214bfda"
server_key_iv: "f62c51009db84c30977408603028de53"
server_key_version: "kuduclusterkey@0"
I20260201 03:04:28.633021 31272 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260201 03:04:28.655203 31289 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:04:28.662925 31272 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260201 03:04:28.663105 31272 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "9d831bbc48344df498a0a939cb6e2792"
format_stamp: "Formatted at 2026-02-01 03:04:28 on dist-test-slave-gf59"
server_key: "d63500b16022654a6ac8f6161214bfda"
server_key_iv: "f62c51009db84c30977408603028de53"
server_key_version: "kuduclusterkey@0"
I20260201 03:04:28.663268 31272 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:04:28.708241 31272 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:04:28.712169 31272 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:04:28.712495 31272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:04:28.713267 31272 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:04:28.714545 31272 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:04:28.714622 31272 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:04:28.714682 31272 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:04:28.714715 31272 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:04:28.726651 31272 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:35873
I20260201 03:04:28.729295 31404 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:35873 every 8 connection(s)
I20260201 03:04:28.730476 31272 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:04:28.732517 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 31272
I20260201 03:04:28.732682 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:04:28.732973 19247 external_mini_cluster.cc:1468] Setting key fc1f2a9b4a084f6040e2dc3c383e95f0
Feb 01 03:04:28 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915068, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
I20260201 03:04:28.748014 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:28.735032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:58793 (local address 127.18.203.254:37471)
0201 03:04:28.736865 (+ 1833us) server_negotiation.cc:207] Beginning negotiation
0201 03:04:28.736870 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:04:28.736884 (+ 14us) server_negotiation.cc:408] Connection header received
0201 03:04:28.736924 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:04:28.736927 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:04:28.736974 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:04:28.737068 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:04:28.738748 (+ 1680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:28.739579 (+ 831us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:04:28.740984 (+ 1405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:28.741172 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:28.743106 (+ 1934us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:04:28.743124 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:04:28.743128 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:04:28.743155 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:04:28.745017 (+ 1862us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:04:28.745632 (+ 615us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:04:28.745635 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:04:28.745638 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:04:28.745690 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:04:28.746053 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:04:28.746057 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:04:28.746058 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:04:28.746243 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:04:28.747566 (+ 1323us) server_negotiation.cc:1036] Waiting for connection context
0201 03:04:28.747686 (+ 120us) server_negotiation.cc:300] Negotiation successful
0201 03:04:28.747819 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1748,"thread_start_us":89,"threads_started":1}
I20260201 03:04:28.748181 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:28.733931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:37471 (local address 127.18.203.193:58793)
0201 03:04:28.735377 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0201 03:04:28.735426 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0201 03:04:28.736421 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:04:28.737298 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:04:28.737310 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:04:28.737723 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:04:28.738564 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:28.738584 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:28.739763 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:04:28.739767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:04:28.740421 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:28.740433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:28.740631 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:28.741339 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:04:28.741375 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:04:28.742942 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:04:28.745150 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:04:28.745156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:04:28.745167 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:04:28.745507 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:04:28.745797 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:04:28.745801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:04:28.745803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:04:28.745937 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:04:28.746375 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:04:28.746381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:04:28.746706 (+ 325us) client_negotiation.cc:769] Sending connection context
0201 03:04:28.746930 (+ 224us) client_negotiation.cc:240] Negotiation successful
0201 03:04:28.747205 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":133,"threads_started":1}
I20260201 03:04:28.749630 31405 heartbeater.cc:344] Connected to a master server at 127.18.203.254:37471
I20260201 03:04:28.749953 31405 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
I20260201 03:04:28.750572 31405 heartbeater.cc:507] Master 127.18.203.254:37471 requested a full tablet report, sending...
I20260201 03:04:28.753623 31190 ts_manager.cc:194] Registered new tserver with Master: 9d831bbc48344df498a0a939cb6e2792 (127.18.203.193:35873)
I20260201 03:04:28.755299 31190 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:58793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Feb 01 03:04:28 dist-test-slave-gf59 krb5kdc[30687](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1769915068, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:04:28.800236 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.203.212:57641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:37471
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:04:28.914659 31413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:04:28.914911 31413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:04:28.914986 31413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:04:28.918380 31413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:04:28.918452 31413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:04:28.918555 31413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:04:28.922747 31413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.203.212:57641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:37471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:04:28.923904 31413 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:04:28.924816 31413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:04:28.932085 31420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:04:28.932638 31419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:04:28.932787 31422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:04:28.935261 31413 server_base.cc:1060] running on GCE node
I20260201 03:04:28.935678 31413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:04:28.936332 31413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:04:28.937537 31413 hybrid_clock.cc:648] HybridClock initialized: now 1769915068937525 us; error 35 us; skew 500 ppm
Feb 01 03:04:28 dist-test-slave-gf59 krb5kdc[30687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915068, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:04:28.940076 31413 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:04:28.941102 31413 webserver.cc:492] Webserver started at http://127.18.203.194:39917/ using document root <none> and password file <none>
I20260201 03:04:28.941628 31413 fs_manager.cc:362] Metadata directory not provided
I20260201 03:04:28.941677 31413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:04:28.941876 31413 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 01 03:04:28 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915068, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
I20260201 03:04:28.954982 31413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "11207a90e14648349496792fb51ba698"
format_stamp: "Formatted at 2026-02-01 03:04:28 on dist-test-slave-gf59"
server_key: "0d8f6f10b3c1ace188bf5ce92cf28b88"
server_key_iv: "17ed5fd0c6703c59ad8044a7a0bcaf19"
server_key_version: "kuduclusterkey@0"
I20260201 03:04:28.955600 31413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "11207a90e14648349496792fb51ba698"
format_stamp: "Formatted at 2026-02-01 03:04:28 on dist-test-slave-gf59"
server_key: "0d8f6f10b3c1ace188bf5ce92cf28b88"
server_key_iv: "17ed5fd0c6703c59ad8044a7a0bcaf19"
server_key_version: "kuduclusterkey@0"
I20260201 03:04:28.965067 31413 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.003s sys 0.001s
I20260201 03:04:28.981703 31429 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:04:28.983165 31413 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260201 03:04:28.983315 31413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "11207a90e14648349496792fb51ba698"
format_stamp: "Formatted at 2026-02-01 03:04:28 on dist-test-slave-gf59"
server_key: "0d8f6f10b3c1ace188bf5ce92cf28b88"
server_key_iv: "17ed5fd0c6703c59ad8044a7a0bcaf19"
server_key_version: "kuduclusterkey@0"
I20260201 03:04:28.983438 31413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:04:29.037171 31413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:04:29.040886 31413 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:04:29.041162 31413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:04:29.041924 31413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:04:29.043109 31413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:04:29.043219 31413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:04:29.043347 31413 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:04:29.043409 31413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:04:29.054068 31413 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:35737
I20260201 03:04:29.054921 31542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:35737 every 8 connection(s)
I20260201 03:04:29.055449 31413 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:04:29.058983 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 31413
I20260201 03:04:29.059127 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:04:29.059423 19247 external_mini_cluster.cc:1468] Setting key 27a5453a99eb86cba29576c306d8a1a2
Feb 01 03:04:29 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915068, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.195@KRBTEST.COM" created.
I20260201 03:04:29.078092 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.060250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.194:46571 (local address 127.18.203.254:37471)
0201 03:04:29.060554 (+ 304us) server_negotiation.cc:207] Beginning negotiation
0201 03:04:29.060558 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:04:29.065786 (+ 5228us) server_negotiation.cc:408] Connection header received
0201 03:04:29.067043 (+ 1257us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:04:29.067047 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:04:29.067111 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:04:29.067210 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:04:29.068956 (+ 1746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.069836 (+ 880us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.070802 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.071005 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.073790 (+ 2785us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:04:29.073813 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:04:29.073815 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:04:29.073842 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:04:29.075455 (+ 1613us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:04:29.076248 (+ 793us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:04:29.076252 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:04:29.076254 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:04:29.076313 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:04:29.076727 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:04:29.076731 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:04:29.076733 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:04:29.076941 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:04:29.077041 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0201 03:04:29.077761 (+ 720us) server_negotiation.cc:300] Negotiation successful
0201 03:04:29.077908 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260201 03:04:29.079532 31545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.058923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:37471 (local address 127.18.203.194:46571)
0201 03:04:29.065600 (+ 6677us) negotiation.cc:107] Waiting for socket to connect
0201 03:04:29.065669 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0201 03:04:29.066765 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:04:29.067552 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:04:29.067565 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:04:29.067973 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:04:29.068770 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.068788 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.070071 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.070075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:04:29.070661 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.070672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.071262 (+ 590us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.071959 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:04:29.071998 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:04:29.073629 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:04:29.075761 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:04:29.075768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:04:29.075783 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:04:29.076111 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:04:29.076473 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:04:29.076477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:04:29.076479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:04:29.076604 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:04:29.077103 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:04:29.077111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:04:29.077439 (+ 328us) client_negotiation.cc:769] Sending connection context
0201 03:04:29.078364 (+ 925us) client_negotiation.cc:240] Negotiation successful
0201 03:04:29.078649 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6427,"thread_start_us":129,"threads_started":1}
I20260201 03:04:29.080921 31543 heartbeater.cc:344] Connected to a master server at 127.18.203.254:37471
I20260201 03:04:29.081185 31543 heartbeater.cc:461] Registering TS with master...
I20260201 03:04:29.081838 31543 heartbeater.cc:507] Master 127.18.203.254:37471 requested a full tablet report, sending...
I20260201 03:04:29.082978 31190 ts_manager.cc:194] Registered new tserver with Master: 11207a90e14648349496792fb51ba698 (127.18.203.194:35737)
I20260201 03:04:29.083762 31190 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.194@KRBTEST.COM'} at 127.18.203.194:46571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:04:29.123051 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.203.212:57641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.195:0
--local_ip_for_outbound_sockets=127.18.203.195
--webserver_interface=127.18.203.195
--webserver_port=0
--tserver_master_addrs=127.18.203.254:37471
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Feb 01 03:04:29 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915068, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
W20260201 03:04:29.264048 31550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:04:29.264361 31550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:04:29.264444 31550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:04:29.269165 31550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:04:29.269253 31550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:04:29.269369 31550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.195
I20260201 03:04:29.275244 31550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.203.212:57641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.203.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:37471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.195
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:04:29.276650 31550 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:04:29.277733 31550 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:04:29.285799 31558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:04:29.286027 31556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:04:29.286257 31555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:04:29.286422 31550 server_base.cc:1060] running on GCE node
I20260201 03:04:29.286837 31550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:04:29.287495 31550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:04:29.288775 31550 hybrid_clock.cc:648] HybridClock initialized: now 1769915069288753 us; error 41 us; skew 500 ppm
Feb 01 03:04:29 dist-test-slave-gf59 krb5kdc[30687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915069, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:04:29.292188 31550 init.cc:377] Logged in from keytab as kudu/127.18.203.195@KRBTEST.COM (short username kudu)
I20260201 03:04:29.293445 31550 webserver.cc:492] Webserver started at http://127.18.203.195:38243/ using document root <none> and password file <none>
I20260201 03:04:29.294121 31550 fs_manager.cc:362] Metadata directory not provided
I20260201 03:04:29.294188 31550 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:04:29.294390 31550 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 01 03:04:29 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915069, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
I20260201 03:04:29.317258 31550 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/data/instance:
uuid: "590d938919374a2abd13f4f019490d68"
format_stamp: "Formatted at 2026-02-01 03:04:29 on dist-test-slave-gf59"
server_key: "893ee353e04ec6be6f498af8b2545a2f"
server_key_iv: "f5e4214b9b55ec757baf292241dd30de"
server_key_version: "kuduclusterkey@0"
I20260201 03:04:29.317929 31550 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance:
uuid: "590d938919374a2abd13f4f019490d68"
format_stamp: "Formatted at 2026-02-01 03:04:29 on dist-test-slave-gf59"
server_key: "893ee353e04ec6be6f498af8b2545a2f"
server_key_iv: "f5e4214b9b55ec757baf292241dd30de"
server_key_version: "kuduclusterkey@0"
I20260201 03:04:29.325124 31550 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260201 03:04:29.342609 31565 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:04:29.344381 31550 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260201 03:04:29.344528 31550 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/wal
uuid: "590d938919374a2abd13f4f019490d68"
format_stamp: "Formatted at 2026-02-01 03:04:29 on dist-test-slave-gf59"
server_key: "893ee353e04ec6be6f498af8b2545a2f"
server_key_iv: "f5e4214b9b55ec757baf292241dd30de"
server_key_version: "kuduclusterkey@0"
I20260201 03:04:29.344655 31550 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:04:29.376363 31550 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:04:29.379932 31550 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:04:29.380116 31550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:04:29.380687 31550 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:04:29.381666 31550 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:04:29.381779 31550 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:04:29.381882 31550 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:04:29.381949 31550 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:04:29.394661 31550 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.195:42731
I20260201 03:04:29.399473 31679 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.195:42731 every 8 connection(s)
I20260201 03:04:29.400583 31550 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/data/info.pb
I20260201 03:04:29.403914 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 31550
Feb 01 03:04:29 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915068, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
I20260201 03:04:29.404055 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/wal/instance
I20260201 03:04:29.404353 19247 external_mini_cluster.cc:1468] Setting key a314c979ca64ec944563a0d2987e7005
Feb 01 03:04:29 dist-test-slave-gf59 krb5kdc[30687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915069, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.195@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:04:29.421811 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.404092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:57743 (local address 127.18.203.254:37471)
0201 03:04:29.404226 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0201 03:04:29.404230 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:04:29.404687 (+ 457us) server_negotiation.cc:408] Connection header received
0201 03:04:29.407153 (+ 2466us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:04:29.407157 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:04:29.407212 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:04:29.407261 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:04:29.409295 (+ 2034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.410119 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.411040 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.411232 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.414366 (+ 3134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:04:29.414386 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:04:29.414390 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:04:29.414420 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:04:29.416123 (+ 1703us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:04:29.420063 (+ 3940us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:04:29.420066 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:04:29.420068 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:04:29.420122 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:04:29.420510 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:04:29.420514 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:04:29.420516 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:04:29.420717 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:04:29.420820 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0201 03:04:29.421504 (+ 684us) server_negotiation.cc:300] Negotiation successful
0201 03:04:29.421640 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260201 03:04:29.422587 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.402971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:37471 (local address 127.18.203.195:57743)
0201 03:04:29.404530 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0201 03:04:29.404577 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0201 03:04:29.406916 (+ 2339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:04:29.407715 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:04:29.407727 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:04:29.408076 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:04:29.409137 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.409156 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.410273 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.410277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:04:29.410910 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.410920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.411462 (+ 542us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.412526 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:04:29.412561 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:04:29.414190 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:04:29.419590 (+ 5400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:04:29.419596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:04:29.419632 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:04:29.419951 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:04:29.420232 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:04:29.420236 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:04:29.420238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:04:29.420388 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:04:29.420877 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:04:29.420882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:04:29.421181 (+ 299us) client_negotiation.cc:769] Sending connection context
0201 03:04:29.421431 (+ 250us) client_negotiation.cc:240] Negotiation successful
0201 03:04:29.421717 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":127,"threads_started":1}
I20260201 03:04:29.423782 31680 heartbeater.cc:344] Connected to a master server at 127.18.203.254:37471
I20260201 03:04:29.424046 31680 heartbeater.cc:461] Registering TS with master...
I20260201 03:04:29.424556 31680 heartbeater.cc:507] Master 127.18.203.254:37471 requested a full tablet report, sending...
I20260201 03:04:29.425731 31190 ts_manager.cc:194] Registered new tserver with Master: 590d938919374a2abd13f4f019490d68 (127.18.203.195:42731)
I20260201 03:04:29.426404 31190 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.195@KRBTEST.COM'} at 127.18.203.195:57743
I20260201 03:04:29.429760 19247 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260201 03:04:29.440550 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.432182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.18.203.254:37471)
0201 03:04:29.432304 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0201 03:04:29.432308 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0201 03:04:29.432515 (+ 207us) server_negotiation.cc:408] Connection header received
0201 03:04:29.432699 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:04:29.432703 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:04:29.432767 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:04:29.432855 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:04:29.433928 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.434738 (+ 810us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.435687 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.435844 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.437250 (+ 1406us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:04:29.437267 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:04:29.437270 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:04:29.437299 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:04:29.438973 (+ 1674us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:04:29.439394 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:04:29.439398 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:04:29.439400 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:04:29.439450 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:04:29.439731 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:04:29.439736 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:04:29.439738 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:04:29.439899 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:04:29.440004 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0201 03:04:29.440267 (+ 263us) server_negotiation.cc:300] Negotiation successful
0201 03:04:29.440381 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260201 03:04:29.443769 31190 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260201 03:04:29.447295 31190 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260201 03:04:29.464143 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.457121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:35737 (local address 127.0.0.1:54886)
0201 03:04:29.457538 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0201 03:04:29.457583 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0201 03:04:29.457996 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:04:29.460729 (+ 2733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:04:29.460735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:04:29.460759 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:04:29.461077 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.461086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.462680 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.462684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:04:29.463657 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.463676 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.463757 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.463813 (+ 56us) client_negotiation.cc:769] Sending connection context
0201 03:04:29.463896 (+ 83us) client_negotiation.cc:240] Negotiation successful
0201 03:04:29.463989 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":131,"threads_started":1}
I20260201 03:04:29.465070 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.459328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:35873 (local address 127.0.0.1:53798)
0201 03:04:29.459587 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0201 03:04:29.459601 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:04:29.459733 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:04:29.461189 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:04:29.461193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:04:29.461213 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:04:29.461477 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.461485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.463045 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.463049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:04:29.464754 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.464765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.464848 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.464864 (+ 16us) client_negotiation.cc:769] Sending connection context
0201 03:04:29.464900 (+ 36us) client_negotiation.cc:240] Negotiation successful
0201 03:04:29.464951 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20260201 03:04:29.466056 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.458072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.18.203.194:35737)
0201 03:04:29.460261 (+ 2189us) server_negotiation.cc:207] Beginning negotiation
0201 03:04:29.460267 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:04:29.460291 (+ 24us) server_negotiation.cc:408] Connection header received
0201 03:04:29.460373 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:04:29.460378 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:04:29.460547 (+ 169us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:04:29.461239 (+ 692us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:04:29.461380 (+ 141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.462526 (+ 1146us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.464798 (+ 2272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.465630 (+ 832us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.465767 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0201 03:04:29.465831 (+ 64us) server_negotiation.cc:300] Negotiation successful
0201 03:04:29.465916 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2088,"thread_start_us":115,"threads_started":1}
I20260201 03:04:29.466193 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.459743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.18.203.193:35873)
0201 03:04:29.460020 (+ 277us) server_negotiation.cc:207] Beginning negotiation
0201 03:04:29.460026 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:04:29.460048 (+ 22us) server_negotiation.cc:408] Connection header received
0201 03:04:29.460133 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:04:29.460139 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:04:29.460327 (+ 188us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:04:29.461631 (+ 1304us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:04:29.461765 (+ 134us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.462911 (+ 1146us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.465280 (+ 2369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.465808 (+ 528us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.465914 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0201 03:04:29.465974 (+ 60us) server_negotiation.cc:300] Negotiation successful
0201 03:04:29.466058 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260201 03:04:29.469954 31339 tablet_service.cc:1505] Processing CreateTablet for tablet 10efff182e214cd09c4536e32dffcb68 (DEFAULT_TABLE table=test-table [id=f96fd8560fb34812ab3ce83d8026781c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:04:29.469954 31476 tablet_service.cc:1505] Processing CreateTablet for tablet 10efff182e214cd09c4536e32dffcb68 (DEFAULT_TABLE table=test-table [id=f96fd8560fb34812ab3ce83d8026781c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:04:29.470978 31339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10efff182e214cd09c4536e32dffcb68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:04:29.470978 31476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10efff182e214cd09c4536e32dffcb68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:04:29.478807 31700 tablet_bootstrap.cc:492] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792: Bootstrap starting.
I20260201 03:04:29.480952 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.457257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.195:42731 (local address 127.0.0.1:39156)
0201 03:04:29.457835 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0201 03:04:29.457849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0201 03:04:29.467039 (+ 9190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:04:29.475770 (+ 8731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:04:29.475774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:04:29.475797 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:04:29.476081 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.476090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.479503 (+ 3413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.479507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:04:29.480522 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.480532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.480665 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.480688 (+ 23us) client_negotiation.cc:769] Sending connection context
0201 03:04:29.480735 (+ 47us) client_negotiation.cc:240] Negotiation successful
0201 03:04:29.480804 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":78,"threads_started":1}
I20260201 03:04:29.481730 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.458550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.18.203.195:42731)
0201 03:04:29.458918 (+ 368us) server_negotiation.cc:207] Beginning negotiation
0201 03:04:29.458925 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0201 03:04:29.458950 (+ 25us) server_negotiation.cc:408] Connection header received
0201 03:04:29.467233 (+ 8283us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:04:29.467239 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:04:29.467429 (+ 190us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:04:29.467550 (+ 121us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:04:29.476246 (+ 8696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.477314 (+ 1068us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.480717 (+ 3403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.481318 (+ 601us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.481445 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0201 03:04:29.481517 (+ 72us) server_negotiation.cc:300] Negotiation successful
0201 03:04:29.481613 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":144,"threads_started":1}
I20260201 03:04:29.483829 31700 tablet_bootstrap.cc:654] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792: Neither blocks nor log segments found. Creating new log.
I20260201 03:04:29.484716 31613 tablet_service.cc:1505] Processing CreateTablet for tablet 10efff182e214cd09c4536e32dffcb68 (DEFAULT_TABLE table=test-table [id=f96fd8560fb34812ab3ce83d8026781c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260201 03:04:29.485208 31699 tablet_bootstrap.cc:492] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698: Bootstrap starting.
I20260201 03:04:29.485383 31700 log.cc:826] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792: Log is configured to *not* fsync() on all Append() calls
I20260201 03:04:29.485747 31613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10efff182e214cd09c4536e32dffcb68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:04:29.487993 31699 tablet_bootstrap.cc:654] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698: Neither blocks nor log segments found. Creating new log.
I20260201 03:04:29.489233 31699 log.cc:826] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698: Log is configured to *not* fsync() on all Append() calls
I20260201 03:04:29.491986 31699 tablet_bootstrap.cc:492] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698: No bootstrap required, opened a new log
I20260201 03:04:29.492200 31699 ts_tablet_manager.cc:1403] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260201 03:04:29.495999 31703 tablet_bootstrap.cc:492] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68: Bootstrap starting.
I20260201 03:04:29.496791 31700 tablet_bootstrap.cc:492] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792: No bootstrap required, opened a new log
I20260201 03:04:29.497092 31700 ts_tablet_manager.cc:1403] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260201 03:04:29.498260 31703 tablet_bootstrap.cc:654] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68: Neither blocks nor log segments found. Creating new log.
I20260201 03:04:29.499480 31703 log.cc:826] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68: Log is configured to *not* fsync() on all Append() calls
I20260201 03:04:29.500962 31700 raft_consensus.cc:359] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11207a90e14648349496792fb51ba698" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 35737 } } peers { permanent_uuid: "590d938919374a2abd13f4f019490d68" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 42731 } } peers { permanent_uuid: "9d831bbc48344df498a0a939cb6e2792" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35873 } }
I20260201 03:04:29.501312 31700 raft_consensus.cc:385] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:04:29.501423 31700 raft_consensus.cc:740] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d831bbc48344df498a0a939cb6e2792, State: Initialized, Role: FOLLOWER
I20260201 03:04:29.501461 31703 tablet_bootstrap.cc:492] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68: No bootstrap required, opened a new log
I20260201 03:04:29.501641 31703 ts_tablet_manager.cc:1403] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260201 03:04:29.502064 31700 consensus_queue.cc:260] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11207a90e14648349496792fb51ba698" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 35737 } } peers { permanent_uuid: "590d938919374a2abd13f4f019490d68" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 42731 } } peers { permanent_uuid: "9d831bbc48344df498a0a939cb6e2792" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35873 } }
I20260201 03:04:29.503073 31700 ts_tablet_manager.cc:1434] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260201 03:04:29.503551 31405 heartbeater.cc:499] Master 127.18.203.254:37471 was elected leader, sending a full tablet report...
I20260201 03:04:29.504482 31703 raft_consensus.cc:359] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11207a90e14648349496792fb51ba698" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 35737 } } peers { permanent_uuid: "590d938919374a2abd13f4f019490d68" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 42731 } } peers { permanent_uuid: "9d831bbc48344df498a0a939cb6e2792" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35873 } }
I20260201 03:04:29.506537 31703 raft_consensus.cc:385] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:04:29.506596 31703 raft_consensus.cc:740] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590d938919374a2abd13f4f019490d68, State: Initialized, Role: FOLLOWER
I20260201 03:04:29.507050 31703 consensus_queue.cc:260] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11207a90e14648349496792fb51ba698" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 35737 } } peers { permanent_uuid: "590d938919374a2abd13f4f019490d68" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 42731 } } peers { permanent_uuid: "9d831bbc48344df498a0a939cb6e2792" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35873 } }
I20260201 03:04:29.507706 31680 heartbeater.cc:499] Master 127.18.203.254:37471 was elected leader, sending a full tablet report...
I20260201 03:04:29.509927 31703 ts_tablet_manager.cc:1434] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260201 03:04:29.495432 31699 raft_consensus.cc:359] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11207a90e14648349496792fb51ba698" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 35737 } } peers { permanent_uuid: "590d938919374a2abd13f4f019490d68" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 42731 } } peers { permanent_uuid: "9d831bbc48344df498a0a939cb6e2792" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35873 } }
I20260201 03:04:29.510263 31699 raft_consensus.cc:385] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:04:29.510339 31699 raft_consensus.cc:740] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11207a90e14648349496792fb51ba698, State: Initialized, Role: FOLLOWER
I20260201 03:04:29.510830 31699 consensus_queue.cc:260] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11207a90e14648349496792fb51ba698" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 35737 } } peers { permanent_uuid: "590d938919374a2abd13f4f019490d68" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 42731 } } peers { permanent_uuid: "9d831bbc48344df498a0a939cb6e2792" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35873 } }
I20260201 03:04:29.511595 31543 heartbeater.cc:499] Master 127.18.203.254:37471 was elected leader, sending a full tablet report...
I20260201 03:04:29.511900 31699 ts_tablet_manager.cc:1434] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
W20260201 03:04:29.556977 31544 tablet.cc:2378] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:04:29.586328 31706 raft_consensus.cc:493] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260201 03:04:29.586578 31706 raft_consensus.cc:515] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11207a90e14648349496792fb51ba698" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 35737 } } peers { permanent_uuid: "590d938919374a2abd13f4f019490d68" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 42731 } } peers { permanent_uuid: "9d831bbc48344df498a0a939cb6e2792" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35873 } }
I20260201 03:04:29.588052 31706 leader_election.cc:290] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11207a90e14648349496792fb51ba698 (127.18.203.194:35737), 9d831bbc48344df498a0a939cb6e2792 (127.18.203.193:35873)
I20260201 03:04:29.592494 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.588471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.194:35737 (local address 127.18.203.195:40087)
0201 03:04:29.588684 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0201 03:04:29.588703 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0201 03:04:29.588824 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:04:29.589799 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:04:29.589803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:04:29.589821 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:04:29.590100 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.590107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.591132 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.591137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:04:29.592093 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.592104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.592212 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.592235 (+ 23us) client_negotiation.cc:769] Sending connection context
0201 03:04:29.592285 (+ 50us) client_negotiation.cc:240] Negotiation successful
0201 03:04:29.592347 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260201 03:04:29.593518 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.589344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:40087 (local address 127.18.203.194:35737)
0201 03:04:29.589484 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0201 03:04:29.589489 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:04:29.589504 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:04:29.589555 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:04:29.589558 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:04:29.589605 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:04:29.589684 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:04:29.590232 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.590996 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.592766 (+ 1770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.593239 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.593276 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0201 03:04:29.593325 (+ 49us) server_negotiation.cc:300] Negotiation successful
0201 03:04:29.593387 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260201 03:04:29.593688 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.589497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.195:48403 (local address 127.18.203.193:35873)
0201 03:04:29.589631 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0201 03:04:29.589636 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:04:29.589670 (+ 34us) server_negotiation.cc:408] Connection header received
0201 03:04:29.589722 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:04:29.589726 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:04:29.589779 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:04:29.589857 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0201 03:04:29.590674 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.591448 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.592733 (+ 1285us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.593234 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.593272 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0201 03:04:29.593480 (+ 208us) server_negotiation.cc:300] Negotiation successful
0201 03:04:29.593553 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260201 03:04:29.594092 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.588831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:35873 (local address 127.18.203.195:48403)
0201 03:04:29.589175 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0201 03:04:29.589188 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0201 03:04:29.589267 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:04:29.590223 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:04:29.590226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:04:29.590242 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0201 03:04:29.590530 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.590536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.591589 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.591592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:04:29.592585 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:04:29.592594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.593335 (+ 741us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.593350 (+ 15us) client_negotiation.cc:769] Sending connection context
0201 03:04:29.593889 (+ 539us) client_negotiation.cc:240] Negotiation successful
0201 03:04:29.593950 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":130,"threads_started":1}
I20260201 03:04:29.594380 31497 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10efff182e214cd09c4536e32dffcb68" candidate_uuid: "590d938919374a2abd13f4f019490d68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11207a90e14648349496792fb51ba698" is_pre_election: true
I20260201 03:04:29.594698 31497 raft_consensus.cc:2468] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 590d938919374a2abd13f4f019490d68 in term 0.
I20260201 03:04:29.595237 31566 leader_election.cc:304] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11207a90e14648349496792fb51ba698, 590d938919374a2abd13f4f019490d68; no voters:
I20260201 03:04:29.595764 31359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10efff182e214cd09c4536e32dffcb68" candidate_uuid: "590d938919374a2abd13f4f019490d68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d831bbc48344df498a0a939cb6e2792" is_pre_election: true
I20260201 03:04:29.596019 31359 raft_consensus.cc:2468] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 590d938919374a2abd13f4f019490d68 in term 0.
I20260201 03:04:29.596707 31706 raft_consensus.cc:2804] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260201 03:04:29.596788 31706 raft_consensus.cc:493] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260201 03:04:29.596834 31706 raft_consensus.cc:3060] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:04:29.598097 31706 raft_consensus.cc:515] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11207a90e14648349496792fb51ba698" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 35737 } } peers { permanent_uuid: "590d938919374a2abd13f4f019490d68" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 42731 } } peers { permanent_uuid: "9d831bbc48344df498a0a939cb6e2792" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35873 } }
I20260201 03:04:29.598497 31706 leader_election.cc:290] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [CANDIDATE]: Term 1 election: Requested vote from peers 11207a90e14648349496792fb51ba698 (127.18.203.194:35737), 9d831bbc48344df498a0a939cb6e2792 (127.18.203.193:35873)
I20260201 03:04:29.598961 31497 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10efff182e214cd09c4536e32dffcb68" candidate_uuid: "590d938919374a2abd13f4f019490d68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11207a90e14648349496792fb51ba698"
I20260201 03:04:29.599093 31497 raft_consensus.cc:3060] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:04:29.599735 31359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10efff182e214cd09c4536e32dffcb68" candidate_uuid: "590d938919374a2abd13f4f019490d68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d831bbc48344df498a0a939cb6e2792"
I20260201 03:04:29.599855 31359 raft_consensus.cc:3060] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:04:29.601024 31359 raft_consensus.cc:2468] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 590d938919374a2abd13f4f019490d68 in term 1.
I20260201 03:04:29.601024 31497 raft_consensus.cc:2468] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 590d938919374a2abd13f4f019490d68 in term 1.
I20260201 03:04:29.601364 31567 leader_election.cc:304] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 590d938919374a2abd13f4f019490d68, 9d831bbc48344df498a0a939cb6e2792; no voters:
I20260201 03:04:29.601719 31706 raft_consensus.cc:2804] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:04:29.601995 31706 raft_consensus.cc:697] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 1 LEADER]: Becoming Leader. State: Replica: 590d938919374a2abd13f4f019490d68, State: Running, Role: LEADER
I20260201 03:04:29.602376 31706 consensus_queue.cc:237] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11207a90e14648349496792fb51ba698" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 35737 } } peers { permanent_uuid: "590d938919374a2abd13f4f019490d68" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 42731 } } peers { permanent_uuid: "9d831bbc48344df498a0a939cb6e2792" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35873 } }
I20260201 03:04:29.606056 31190 catalog_manager.cc:5654] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 590d938919374a2abd13f4f019490d68 (127.18.203.195). New cstate: current_term: 1 leader_uuid: "590d938919374a2abd13f4f019490d68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11207a90e14648349496792fb51ba698" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 35737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "590d938919374a2abd13f4f019490d68" member_type: VOTER last_known_addr { host: "127.18.203.195" port: 42731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d831bbc48344df498a0a939cb6e2792" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35873 } health_report { overall_health: UNKNOWN } } }
I20260201 03:04:29.654112 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:04:29.649721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.18.203.195:42731)
0201 03:04:29.650076 (+ 355us) server_negotiation.cc:207] Beginning negotiation
0201 03:04:29.650081 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0201 03:04:29.650096 (+ 15us) server_negotiation.cc:408] Connection header received
0201 03:04:29.650191 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:04:29.650195 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:04:29.650261 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:04:29.650359 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
0201 03:04:29.650902 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.651762 (+ 860us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:04:29.652715 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:04:29.652928 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:04:29.653052 (+ 124us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0201 03:04:29.653681 (+ 629us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0201 03:04:29.653819 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0201 03:04:29.653904 (+ 85us) server_negotiation.cc:300] Negotiation successful
0201 03:04:29.653967 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260201 03:04:29.660619 31359 raft_consensus.cc:1275] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792 [term 1 FOLLOWER]: Refusing update from remote peer 590d938919374a2abd13f4f019490d68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:04:29.660715 31497 raft_consensus.cc:1275] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698 [term 1 FOLLOWER]: Refusing update from remote peer 590d938919374a2abd13f4f019490d68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260201 03:04:29.661286 31706 consensus_queue.cc:1048] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d831bbc48344df498a0a939cb6e2792" member_type: VOTER last_known_addr { host: "127.18.203.193" port: 35873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260201 03:04:29.661712 31706 consensus_queue.cc:1048] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11207a90e14648349496792fb51ba698" member_type: VOTER last_known_addr { host: "127.18.203.194" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260201 03:04:29.665342 31681 tablet.cc:2378] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260201 03:04:29.671478 31712 mvcc.cc:204] Tried to move back new op lower bound from 7249572125322272768 to 7249572125094875136. Current Snapshot: MvccSnapshot[applied={T|T < 7249572125322272768}]
I20260201 03:04:29.673568 31713 mvcc.cc:204] Tried to move back new op lower bound from 7249572125322272768 to 7249572125094875136. Current Snapshot: MvccSnapshot[applied={T|T < 7249572125322272768}]
I20260201 03:04:29.675505 31714 mvcc.cc:204] Tried to move back new op lower bound from 7249572125322272768 to 7249572125094875136. Current Snapshot: MvccSnapshot[applied={T|T < 7249572125322272768}]
I20260201 03:04:29.684428 31189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:04:29.684609 31189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260201 03:04:29.687505 31189 catalog_manager.cc:5941] T 00000000000000000000000000000000 P bead5d100ab647c9adc567a921aaf355: Sending DeleteTablet for 3 replicas of tablet 10efff182e214cd09c4536e32dffcb68
I20260201 03:04:29.689241 31476 tablet_service.cc:1552] Processing DeleteTablet for tablet 10efff182e214cd09c4536e32dffcb68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:04:29 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:54886
I20260201 03:04:29.689745 31722 tablet_replica.cc:333] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698: stopping tablet replica
I20260201 03:04:29.689726 31613 tablet_service.cc:1552] Processing DeleteTablet for tablet 10efff182e214cd09c4536e32dffcb68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:04:29 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:39156
I20260201 03:04:29.689996 31722 raft_consensus.cc:2243] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:04:29.690162 31339 tablet_service.cc:1552] Processing DeleteTablet for tablet 10efff182e214cd09c4536e32dffcb68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-01 03:04:29 UTC) from {username='kudu', principal='kudu/127.18.203.254@KRBTEST.COM'} at 127.0.0.1:53798
I20260201 03:04:29.690232 31722 raft_consensus.cc:2272] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:04:29.690575 31724 tablet_replica.cc:333] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792: stopping tablet replica
I20260201 03:04:29.690829 31724 raft_consensus.cc:2243] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260201 03:04:29.691107 31724 raft_consensus.cc:2272] T 10efff182e214cd09c4536e32dffcb68 P 9d831bbc48344df498a0a939cb6e2792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:04:29.691138 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 31272
I20260201 03:04:29.693568 31722 ts_tablet_manager.cc:1916] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260201 03:04:29.695523 31723 tablet_replica.cc:333] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68: stopping tablet replica
I20260201 03:04:29.695791 31723 raft_consensus.cc:2243] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 1 LEADER]: Raft consensus shutting down.
I20260201 03:04:29.696075 31722 ts_tablet_manager.cc:1929] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:04:29.696156 31722 log.cc:1199] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-1/wal/wals/10efff182e214cd09c4536e32dffcb68
I20260201 03:04:29.696187 31723 raft_consensus.cc:2272] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260201 03:04:29.696583 31722 ts_tablet_manager.cc:1950] T 10efff182e214cd09c4536e32dffcb68 P 11207a90e14648349496792fb51ba698: Deleting consensus metadata
I20260201 03:04:29.697767 31174 catalog_manager.cc:4985] TS 11207a90e14648349496792fb51ba698 (127.18.203.194:35737): tablet 10efff182e214cd09c4536e32dffcb68 (table test-table [id=f96fd8560fb34812ab3ce83d8026781c]) successfully deleted
I20260201 03:04:29.699118 31723 ts_tablet_manager.cc:1916] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260201 03:04:29.699496 31175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.203.193:35873 (error 108)
W20260201 03:04:29.700450 31694 negotiation.cc:337] Failed RPC negotiation. Trace:
0201 03:04:29.700185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.193:35873 (local address 127.0.0.1:53814)
0201 03:04:29.700307 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0201 03:04:29.700361 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.193:35873: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
W20260201 03:04:29.700557 31175 catalog_manager.cc:4712] TS 9d831bbc48344df498a0a939cb6e2792 (127.18.203.193:35873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10efff182e214cd09c4536e32dffcb68: Network error: Client connection negotiation failed: client connection to 127.18.203.193:35873: connect: Connection reset by peer (error 104)
I20260201 03:04:29.700923 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 31413
I20260201 03:04:29.706127 31723 ts_tablet_manager.cc:1929] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260201 03:04:29.706281 31723 log.cc:1199] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68: Deleting WAL directory at /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769914928154323-19247-0/minicluster-data/ts-2/wal/wals/10efff182e214cd09c4536e32dffcb68
I20260201 03:04:29.706691 31723 ts_tablet_manager.cc:1950] T 10efff182e214cd09c4536e32dffcb68 P 590d938919374a2abd13f4f019490d68: Deleting consensus metadata
I20260201 03:04:29.707973 31177 catalog_manager.cc:4985] TS 590d938919374a2abd13f4f019490d68 (127.18.203.195:42731): tablet 10efff182e214cd09c4536e32dffcb68 (table test-table [id=f96fd8560fb34812ab3ce83d8026781c]) successfully deleted
I20260201 03:04:29.709178 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 31550
I20260201 03:04:29.717986 19247 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskyeWP68/build/debug/bin/kudu with pid 31158
I20260201 03:04:29.724406 19247 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260201 03:04:30.058104 19247 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260201 03:04:30.058249 19247 mini_ranger.cc:66] Stopping Ranger...
I20260201 03:04:30.402007 19247 mini_ranger.cc:68] Stopped Ranger
2026-02-01 03:04:30.402 UTC [30718] LOG: received smart shutdown request
2026-02-01 03:04:30.405 UTC [30718] LOG: background worker "logical replication launcher" (PID 30726) exited with exit code 1
2026-02-01 03:04:30.405 UTC [30721] LOG: shutting down
2026-02-01 03:04:30.407 UTC [30721] LOG: checkpoint starting: shutdown immediate
2026-02-01 03:04:31.203 UTC [30721] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.041 s, sync=0.751 s, total=0.798 s; sync files=1108, longest=0.005 s, average=0.001 s; distance=12074 kB, estimate=12074 kB; lsn=0/2045130, redo lsn=0/2045130
2026-02-01 03:04:31.211 UTC [30718] LOG: database system is shut down
2026-02-01T03:04:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46697 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 01 03:04:31 dist-test-slave-gf59 krb5kdc[31739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 01 03:04:31 dist-test-slave-gf59 krb5kdc[31739](info): set up 2 sockets
Feb 01 03:04:31 dist-test-slave-gf59 krb5kdc[31739](info): commencing operation
krb5kdc: starting...
W20260201 03:04:33.151768 19247 mini_kdc.cc:121] Time spent starting KDC: real 1.814s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/test-user.keytab.
Feb 01 03:04:33 dist-test-slave-gf59 krb5kdc[31739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-01T03:04:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-01T03:04:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/rangeradmin_127.18.203.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/rangeradmin_127.18.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/rangerlookup_127.18.203.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/rangerlookup_127.18.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/HTTP_127.18.203.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/HTTP_127.18.203.212@KRBTEST.COM.keytab.
I20260201 03:04:33.311340 19247 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyeWP68/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/postgres -l logfile start
2026-02-01 03:04:35.618 UTC [31770] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-02-01 03:04:35.618 UTC [31770] LOG: listening on IPv4 address "127.18.203.212", port 44441
2026-02-01 03:04:35.621 UTC [31770] LOG: listening on Unix socket "/tmp/.s.PGSQL.44441"
2026-02-01 03:04:35.626 UTC [31775] LOG: database system was shut down at 2026-02-01 03:04:34 UTC
2026-02-01 03:04:35.630 UTC [31770] LOG: database system is ready to accept connections
I20260201 03:04:37.375631 19247 mini_postgres.cc:96] Postgres bound to 44441
2026-02-01 03:04:37.382 UTC [31781] FATAL: database "slave" does not exist
127.18.203.212:44441 - accepting connections
I20260201 03:04:37.382961 19247 mini_ranger.cc:161] Starting Ranger...
I20260201 03:04:37.400205 19247 mini_ranger.cc:84] Created miniranger Postgres user
I20260201 03:04:37.463598 19247 mini_ranger.cc:87] Created ranger Postgres database
I20260201 03:04:37.463734 19247 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-admin
2026-02-01 03:04:37,725 [I] DB FLAVOR :POSTGRES
2026-02-01 03:04:37,725 [I] --------- Verifying Ranger DB connection ---------
2026-02-01 03:04:37,726 [I] Checking connection..
2026-02-01 03:04:37,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-02-01 03:04:38,019 [I] Checking connection passed.
2026-02-01 03:04:38,019 [I] --------- Verifying version history table ---------
2026-02-01 03:04:38,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-01 03:04:38,308 [I] Table x_db_version_h does not exist in database ranger
2026-02-01 03:04:38,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-01 03:04:38,596 [I] Table x_db_version_h does not exist in database ranger
2026-02-01 03:04:38,596 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-02-01 03:04:38,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-02-01 03:04:38.891 UTC [31848] WARNING: there is no transaction in progress
2026-02-01 03:04:38,898 [I] create_dbversion_catalog.sql file imported successfully
2026-02-01 03:04:38,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-01 03:04:39,192 [I] Table x_db_version_h already exists in database 'ranger'
2026-02-01 03:04:39,193 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-01 03:04:39,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-02-01 03:04:39,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-02-01 03:04:39,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gf59.c.gcp-upstream.internal','N') ;"
2026-02-01 03:04:40,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-01 03:04:40,358 [I] Table x_portal_user does not exist in database ranger
2026-02-01 03:04:40,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-01 03:04:40,671 [I] Table x_policy_ref_group does not exist in database ranger
2026-02-01 03:04:40,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-01 03:04:40,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-01 03:04:41,246 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-02-01 03:04:41,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-02-01 03:04:42.033 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.046 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.058 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.068 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.078 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.093 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.139 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.143 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.153 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.162 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.170 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.179 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.186 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.190 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.193 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42.196 UTC [31965] WARNING: there is no transaction in progress
2026-02-01 03:04:42,545 [I] ranger_core_db_postgres.sql file imported successfully
2026-02-01 03:04:42,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-02-01 03:04:42,842 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-02-01 03:04:42,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-01 03:04:43,154 [I] Table x_portal_user already exists in database 'ranger'
2026-02-01 03:04:43,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-01 03:04:43,450 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-02-01 03:04:43,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-01 03:04:43,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-01 03:04:44,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gf59.c.gcp-upstream.internal';"
2026-02-01 03:04:44,333 [I] CORE_DB_SCHEMA import status has been updated
2026-02-01 03:04:44,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-02-01 03:04:44,630 [I] DB_PATCHES have already been applied
I20260201 03:04:44.635773 19247 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyeWP68/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260201 03:04:44.635843 19247 mini_ranger.cc:193] Using host: 127.18.203.212
I20260201 03:04:44.639437 19247 mini_ranger.cc:227] Ranger admin URL: http://127.18.203.212:34205
Feb 01, 2026 3:04:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 01, 2026 3:04:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 01, 2026 3:04:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-admin/ews/webapp
Feb 01, 2026 3:04:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Feb 01, 2026 3:04:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-admin/ews/webapp] .....
Feb 01, 2026 3:04:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-admin/ews/webapp].
Feb 01, 2026 3:04:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260201 03:04:46.658174 19247 mini_ranger.cc:160] Time spent starting Ranger: real 9.275s user 0.000s sys 0.011s
Feb 01, 2026 3:05:04 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Feb 01, 2026 3:05:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
Feb 01, 2026 3:05:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Feb 01, 2026 3:05:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Feb 01, 2026 3:05:04 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Feb 01, 2026 3:05:05 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260201 03:05:06.019300 19247 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/rangerkms_127.18.203.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/rangerkms_127.18.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.203.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.203.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/HTTP_127.18.203.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.203.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/HTTP_127.18.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Feb 01 03:05:06 dist-test-slave-gf59 krb5kdc[31739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915106, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260201 03:05:06.105640 19247 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260201 03:05:06.116631 19247 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260201 03:05:06.217307 19247 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260201 03:05:06.217399 19247 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-kms
I20260201 03:05:06.217410 19247 mini_ranger_kms.cc:227] Using postgres at 127.18.203.212:44441
2026-02-01 03:05:06,379 [I] DB FLAVOR :POSTGRES
2026-02-01 03:05:06,380 [I] --------- Verifying Ranger DB connection ---------
2026-02-01 03:05:06,380 [I] Checking connection
2026-02-01 03:05:06,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-02-01 03:05:06,706 [I] connection success
2026-02-01 03:05:06,706 [I] --------- Verifying Ranger DB tables ---------
2026-02-01 03:05:06,706 [I] Verifying table ranger_masterkey in database rangerkms
2026-02-01 03:05:06,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-02-01 03:05:07,026 [I] Table ranger_masterkey does not exist in database rangerkms
2026-02-01 03:05:07,026 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-01 03:05:07,026 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-02-01 03:05:07,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.203.212:44441/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-02-01 03:05:07,340 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260201 03:05:07.801566 19247 mini_ranger_kms.cc:310] Created kms service
I20260201 03:05:08.053468 19247 mini_ranger_kms.cc:326] Created kudu user
I20260201 03:05:08.096774 19247 mini_ranger_kms.cc:343] Created rangerkms user
Feb 01 03:05:08 dist-test-slave-gf59 krb5kdc[31739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915106, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
I20260201 03:05:08.477964 19247 mini_ranger_kms.cc:384] Added ranger policy
I20260201 03:05:08.478226 19247 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyeWP68/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyeWP68/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260201 03:05:08.478255 19247 mini_ranger_kms.cc:240] Using host: 127.18.203.212
I20260201 03:05:08.482583 19247 mini_ranger_kms.cc:276] Ranger KMS PID: 32157
I20260201 03:05:08.482707 19247 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.203.212:58435
Feb 01, 2026 3:05:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 01, 2026 3:05:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 01, 2026 3:05:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Feb 01, 2026 3:05:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-kms/ews/webapp] .....
Feb 01, 2026 3:05:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-kms/ews/webapp].
Feb 01, 2026 3:05:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260201 03:05:10.445791 19247 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.340s user 0.000s sys 0.146s
I20260201 03:05:10.445935 19247 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260201 03:05:10.446009 19247 mini_ranger_kms.cc:401] 127.18.203.212:58435/kms/v1/keys
Feb 01 03:05:10 dist-test-slave-gf59 krb5kdc[31739](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1769915110, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.203.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Feb 01 03:05:11 dist-test-slave-gf59 krb5kdc[31739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915110, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.203.212@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
Feb 01 03:05:11 dist-test-slave-gf59 krb5kdc[31739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915110, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.203.212@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
Feb 01 03:05:13 dist-test-slave-gf59 krb5kdc[31739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:05:13.578279 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:33549
--webserver_interface=127.18.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.203.254:33549
--ranger_config_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.203.212:58435/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:05:13.691895 32210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:05:13.692166 32210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:05:13.692221 32210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:05:13.695663 32210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260201 03:05:13.695751 32210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:05:13.695775 32210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:05:13.695792 32210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260201 03:05:13.695814 32210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260201 03:05:13.700521 32210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.203.212:58435/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.203.254:33549
--ranger_config_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.203.254:33549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.203.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:05:13.701722 32210 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:05:13.702642 32210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:05:13.708920 32215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:05:13.709005 32216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:05:13.709110 32218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:05:13.709203 32210 server_base.cc:1060] running on GCE node
I20260201 03:05:13.709903 32210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:05:13.711161 32210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:05:13.712358 32210 hybrid_clock.cc:648] HybridClock initialized: now 1769915113712344 us; error 33 us; skew 500 ppm
Feb 01 03:05:13 dist-test-slave-gf59 krb5kdc[31739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915113, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:05:13.715689 32210 init.cc:377] Logged in from keytab as kudu/127.18.203.254@KRBTEST.COM (short username kudu)
I20260201 03:05:13.717139 32210 webserver.cc:492] Webserver started at http://127.18.203.254:34915/ using document root <none> and password file <none>
I20260201 03:05:13.717835 32210 fs_manager.cc:362] Metadata directory not provided
I20260201 03:05:13.717908 32210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:05:13.718163 32210 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 01 03:05:13 dist-test-slave-gf59 krb5kdc[31739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915113, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.254@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
I20260201 03:05:13.771346 32210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/data/instance:
uuid: "a2983c4fb01a48498cdfd84c6f069fe6"
format_stamp: "Formatted at 2026-02-01 03:05:13 on dist-test-slave-gf59"
server_key: "a76cb03a57b196a0cb549eb0126e182b"
server_key_iv: "1ee00b7da523841c4bccf65c48d4d051"
server_key_version: "kuduclusterkey@0"
I20260201 03:05:13.772109 32210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/wal/instance:
uuid: "a2983c4fb01a48498cdfd84c6f069fe6"
format_stamp: "Formatted at 2026-02-01 03:05:13 on dist-test-slave-gf59"
server_key: "a76cb03a57b196a0cb549eb0126e182b"
server_key_iv: "1ee00b7da523841c4bccf65c48d4d051"
server_key_version: "kuduclusterkey@0"
I20260201 03:05:13.776749 32210 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260201 03:05:13.793705 32225 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:05:13.795154 32210 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260201 03:05:13.795313 32210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/wal
uuid: "a2983c4fb01a48498cdfd84c6f069fe6"
format_stamp: "Formatted at 2026-02-01 03:05:13 on dist-test-slave-gf59"
server_key: "a76cb03a57b196a0cb549eb0126e182b"
server_key_iv: "1ee00b7da523841c4bccf65c48d4d051"
server_key_version: "kuduclusterkey@0"
I20260201 03:05:13.795455 32210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:05:13.819676 32210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:05:13.823016 32210 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:05:13.823208 32210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:05:13.830993 32210 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.254:33549
I20260201 03:05:13.831022 32277 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.254:33549 every 8 connection(s)
I20260201 03:05:13.832288 32210 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/data/info.pb
I20260201 03:05:13.835211 32278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260201 03:05:13.836483 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 32210
I20260201 03:05:13.836623 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/wal/instance
I20260201 03:05:13.836928 19247 external_mini_cluster.cc:1468] Setting key 8d469a107d9bbc8ae17eb49a38443201
I20260201 03:05:13.841444 32278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6: Bootstrap starting.
I20260201 03:05:13.844013 32278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6: Neither blocks nor log segments found. Creating new log.
I20260201 03:05:13.844789 32278 log.cc:826] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6: Log is configured to *not* fsync() on all Append() calls
Feb 01 03:05:13 dist-test-slave-gf59 krb5kdc[31739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:05:13.847023 32278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6: No bootstrap required, opened a new log
I20260201 03:05:13.850431 32278 raft_consensus.cc:359] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2983c4fb01a48498cdfd84c6f069fe6" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 33549 } }
I20260201 03:05:13.850678 32278 raft_consensus.cc:385] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260201 03:05:13.850768 32278 raft_consensus.cc:740] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2983c4fb01a48498cdfd84c6f069fe6, State: Initialized, Role: FOLLOWER
I20260201 03:05:13.851310 32281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:13.838356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.18.203.254:33549)
0201 03:05:13.838819 (+ 463us) server_negotiation.cc:207] Beginning negotiation
0201 03:05:13.838832 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0201 03:05:13.838875 (+ 43us) server_negotiation.cc:408] Connection header received
0201 03:05:13.839554 (+ 679us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:05:13.839581 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:05:13.840015 (+ 434us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:05:13.840390 (+ 375us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:05:13.841488 (+ 1098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:05:13.842627 (+ 1139us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:05:13.843670 (+ 1043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:05:13.843951 (+ 281us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:05:13.846463 (+ 2512us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:05:13.846483 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:05:13.846494 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:05:13.846529 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:05:13.848772 (+ 2243us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:05:13.849190 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:05:13.849195 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:05:13.849200 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:05:13.849267 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:05:13.849484 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:05:13.849489 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:05:13.849490 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:05:13.849849 (+ 359us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:05:13.850058 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
0201 03:05:13.850330 (+ 272us) server_negotiation.cc:300] Negotiation successful
0201 03:05:13.850579 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":138,"threads_started":1}
I20260201 03:05:13.851294 32278 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2983c4fb01a48498cdfd84c6f069fe6" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 33549 } }
I20260201 03:05:13.851460 32278 raft_consensus.cc:399] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260201 03:05:13.851552 32278 raft_consensus.cc:493] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260201 03:05:13.851696 32278 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [term 0 FOLLOWER]: Advancing to term 1
I20260201 03:05:13.852798 32278 raft_consensus.cc:515] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2983c4fb01a48498cdfd84c6f069fe6" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 33549 } }
I20260201 03:05:13.853205 32278 leader_election.cc:304] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2983c4fb01a48498cdfd84c6f069fe6; no voters:
I20260201 03:05:13.853608 32278 leader_election.cc:290] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260201 03:05:13.853704 32283 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260201 03:05:13.853909 32283 raft_consensus.cc:697] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [term 1 LEADER]: Becoming Leader. State: Replica: a2983c4fb01a48498cdfd84c6f069fe6, State: Running, Role: LEADER
I20260201 03:05:13.854256 32283 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2983c4fb01a48498cdfd84c6f069fe6" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 33549 } }
I20260201 03:05:13.854822 32278 sys_catalog.cc:565] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260201 03:05:13.855341 32283 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2983c4fb01a48498cdfd84c6f069fe6. Latest consensus state: current_term: 1 leader_uuid: "a2983c4fb01a48498cdfd84c6f069fe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2983c4fb01a48498cdfd84c6f069fe6" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 33549 } } }
I20260201 03:05:13.855777 32284 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2983c4fb01a48498cdfd84c6f069fe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2983c4fb01a48498cdfd84c6f069fe6" member_type: VOTER last_known_addr { host: "127.18.203.254" port: 33549 } } }
I20260201 03:05:13.855942 32284 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [sys.catalog]: This master's current role is: LEADER
I20260201 03:05:13.856488 32283 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6 [sys.catalog]: This master's current role is: LEADER
I20260201 03:05:13.857224 32278 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260201 03:05:13.858178 32292 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260201 03:05:13.863541 32292 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260201 03:05:13.873831 32292 catalog_manager.cc:1357] Generated new cluster ID: 0558fddd79524143900c1f221f0f82f6
I20260201 03:05:13.874045 32292 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260201 03:05:13.892290 32292 catalog_manager.cc:1380] Generated new certificate authority record
I20260201 03:05:13.893779 32292 catalog_manager.cc:1514] Loading token signing keys...
I20260201 03:05:13.905119 32292 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a2983c4fb01a48498cdfd84c6f069fe6: Generated new TSK 0
I20260201 03:05:13.906263 32292 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab.
I20260201 03:05:14.606411 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.203.212:58435/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.193:0
--local_ip_for_outbound_sockets=127.18.203.193
--webserver_interface=127.18.203.193
--webserver_port=0
--tserver_master_addrs=127.18.203.254:33549
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260201 03:05:14.737560 32324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:05:14.737815 32324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:05:14.737877 32324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:05:14.741338 32324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:05:14.741406 32324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:05:14.741494 32324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.193
I20260201 03:05:14.745877 32324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.203.212:58435/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.203.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.193
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:05:14.746985 32324 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:05:14.748010 32324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:05:14.756561 32332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:05:14.756913 32334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:05:14.757015 32331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:05:14.757925 32324 server_base.cc:1060] running on GCE node
I20260201 03:05:14.758388 32324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:05:14.759133 32324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:05:14.760320 32324 hybrid_clock.cc:648] HybridClock initialized: now 1769915114760296 us; error 44 us; skew 500 ppm
Feb 01 03:05:14 dist-test-slave-gf59 krb5kdc[31739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915114, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:05:14.763947 32324 init.cc:377] Logged in from keytab as kudu/127.18.203.193@KRBTEST.COM (short username kudu)
I20260201 03:05:14.765355 32324 webserver.cc:492] Webserver started at http://127.18.203.193:36397/ using document root <none> and password file <none>
I20260201 03:05:14.766121 32324 fs_manager.cc:362] Metadata directory not provided
I20260201 03:05:14.766212 32324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:05:14.766446 32324 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 01 03:05:14 dist-test-slave-gf59 krb5kdc[31739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915114, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
I20260201 03:05:14.784102 32324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/data/instance:
uuid: "adaf6b02762e474a8f5b4d95f9bb88c5"
format_stamp: "Formatted at 2026-02-01 03:05:14 on dist-test-slave-gf59"
server_key: "66b87d4fe6f3e30e5608ff801376800e"
server_key_iv: "f280a5d3dc5e693af79739810f6097e4"
server_key_version: "kuduclusterkey@0"
I20260201 03:05:14.784919 32324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance:
uuid: "adaf6b02762e474a8f5b4d95f9bb88c5"
format_stamp: "Formatted at 2026-02-01 03:05:14 on dist-test-slave-gf59"
server_key: "66b87d4fe6f3e30e5608ff801376800e"
server_key_iv: "f280a5d3dc5e693af79739810f6097e4"
server_key_version: "kuduclusterkey@0"
I20260201 03:05:14.789939 32324 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260201 03:05:14.811501 32341 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:05:14.813009 32324 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260201 03:05:14.813149 32324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/wal
uuid: "adaf6b02762e474a8f5b4d95f9bb88c5"
format_stamp: "Formatted at 2026-02-01 03:05:14 on dist-test-slave-gf59"
server_key: "66b87d4fe6f3e30e5608ff801376800e"
server_key_iv: "f280a5d3dc5e693af79739810f6097e4"
server_key_version: "kuduclusterkey@0"
I20260201 03:05:14.813271 32324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:05:14.851008 32324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:05:14.854874 32324 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:05:14.855180 32324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:05:14.855997 32324 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:05:14.857214 32324 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:05:14.857326 32324 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:05:14.857450 32324 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:05:14.857627 32324 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:05:14.869707 32324 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.193:41401
I20260201 03:05:14.870110 32456 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.193:41401 every 8 connection(s)
I20260201 03:05:14.870913 32324 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/data/info.pb
I20260201 03:05:14.876482 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 32324
I20260201 03:05:14.876633 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-0/wal/instance
I20260201 03:05:14.876929 19247 external_mini_cluster.cc:1468] Setting key 4c925765ccd9c9247c22d5aa395caa24
Feb 01 03:05:14 dist-test-slave-gf59 krb5kdc[31739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915114, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.193@KRBTEST.COM for kudu/127.18.203.254@KRBTEST.COM
I20260201 03:05:14.885982 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:14.873808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.193:48657)
0201 03:05:14.874280 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:14.874328 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0201 03:05:14.875185 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0201 03:05:14.875768 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0201 03:05:14.875778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0201 03:05:14.876077 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0201 03:05:14.876837 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:05:14.876855 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:05:14.877894 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0201 03:05:14.877898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0201 03:05:14.878450 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0201 03:05:14.878460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0201 03:05:14.878682 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:05:14.879395 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0201 03:05:14.879431 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0201 03:05:14.881043 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0201 03:05:14.883047 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:05:14.883053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:05:14.883066 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:05:14.883309 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:05:14.883589 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0201 03:05:14.883592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0201 03:05:14.883594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0201 03:05:14.883763 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0201 03:05:14.884188 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0201 03:05:14.884195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0201 03:05:14.884512 (+ 317us) client_negotiation.cc:769] Sending connection context
0201 03:05:14.884734 (+ 222us) client_negotiation.cc:240] Negotiation successful
0201 03:05:14.885014 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":101,"threads_started":1}
I20260201 03:05:14.886716 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:14.873464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.203.193:48657 (local address 127.18.203.254:33549)
0201 03:05:14.874853 (+ 1389us) server_negotiation.cc:207] Beginning negotiation
0201 03:05:14.874859 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0201 03:05:14.874878 (+ 19us) server_negotiation.cc:408] Connection header received
0201 03:05:14.875402 (+ 524us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0201 03:05:14.875406 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0201 03:05:14.875468 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0201 03:05:14.875552 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0201 03:05:14.877037 (+ 1485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:05:14.877755 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0201 03:05:14.879037 (+ 1282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0201 03:05:14.879225 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0201 03:05:14.881233 (+ 2008us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0201 03:05:14.881257 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0201 03:05:14.881261 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0201 03:05:14.881292 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0201 03:05:14.882922 (+ 1630us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:05:14.883407 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:05:14.883411 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:05:14.883413 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:05:14.883471 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0201 03:05:14.883859 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0201 03:05:14.883862 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0201 03:05:14.883863 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0201 03:05:14.884051 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0201 03:05:14.884134 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0201 03:05:14.886409 (+ 2275us) server_negotiation.cc:300] Negotiation successful
0201 03:05:14.886552 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1289,"thread_start_us":90,"threads_started":1}
I20260201 03:05:14.888062 32457 heartbeater.cc:344] Connected to a master server at 127.18.203.254:33549
I20260201 03:05:14.888340 32457 heartbeater.cc:461] Registering TS with master...
I20260201 03:05:14.889045 32457 heartbeater.cc:507] Master 127.18.203.254:33549 requested a full tablet report, sending...
I20260201 03:05:14.890995 32241 ts_manager.cc:194] Registered new tserver with Master: adaf6b02762e474a8f5b4d95f9bb88c5 (127.18.203.193:41401)
WARNING: no policy specified for kudu/127.18.203.194@KRBTEST.COM; defaulting to no policy
I20260201 03:05:14.892397 32241 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.203.193@KRBTEST.COM'} at 127.18.203.193:48657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.18.203.254 from keytab /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
WARNING: no policy specified for HTTP/127.18.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.203.194@KRBTEST.COM" created.
I20260201 03:05:14.921783 32316 server.cc:273] Received an EOF from the subprocess
F20260201 03:05:14.922631 32302 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7fd737beedcd google::LogMessage::Fail() at ??:0
@ 0x7fd737bf2b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd737bee7cc google::LogMessage::Flush() at ??:0
@ 0x7fd737beff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd7398a62d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7fd7398a30d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7fd7398a7005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x561dd4b56a0a std::function<>::operator()() at ??:0
@ 0x7fd739048b04 kudu::Thread::SuperviseThread() at ??:0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab.
@ 0x7fd7398e26db start_thread at ??:0
I20260201 03:05:14.938328 19247 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskyeWP68/build/debug/bin/kudu
/tmp/dist-test-taskyeWP68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.203.212:58435/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.203.194:0
--local_ip_for_outbound_sockets=127.18.203.194
--webserver_interface=127.18.203.194
--webserver_port=0
--tserver_master_addrs=127.18.203.254:33549
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.203.212:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
@ 0x7fd736da571f clone at ??:0
W20260201 03:05:15.051862 32470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260201 03:05:15.052155 32470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260201 03:05:15.052242 32470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260201 03:05:15.055804 32470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260201 03:05:15.055891 32470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260201 03:05:15.056002 32470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.203.194
I20260201 03:05:15.060531 32470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.203.212:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.203.212:58435/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.203.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.203.254:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.203.194
--log_dir=/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 01 Feb 2026 02:43:14 UTC on 24a791456cd2
build id 10286
I20260201 03:05:15.061682 32470 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260201 03:05:15.062587 32470 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260201 03:05:15.069490 32483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:05:15.069514 32482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260201 03:05:15.069628 32485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260201 03:05:15.069911 32470 server_base.cc:1060] running on GCE node
I20260201 03:05:15.070341 32470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260201 03:05:15.071084 32470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260201 03:05:15.072263 32470 hybrid_clock.cc:648] HybridClock initialized: now 1769915115072243 us; error 39 us; skew 500 ppm
Feb 01 03:05:15 dist-test-slave-gf59 krb5kdc[31739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915115, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260201 03:05:15.075835 32470 init.cc:377] Logged in from keytab as kudu/127.18.203.194@KRBTEST.COM (short username kudu)
I20260201 03:05:15.077303 32470 webserver.cc:492] Webserver started at http://127.18.203.194:39799/ using document root <none> and password file <none>
I20260201 03:05:15.078032 32470 fs_manager.cc:362] Metadata directory not provided
I20260201 03:05:15.078109 32470 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260201 03:05:15.078347 32470 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 01 03:05:15 dist-test-slave-gf59 krb5kdc[31739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769915115, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.203.194@KRBTEST.COM for HTTP/127.18.203.212@KRBTEST.COM
I20260201 03:05:15.093464 32470 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/data/instance:
uuid: "39fbe509bc164eb78223e3fbbc1f82b0"
format_stamp: "Formatted at 2026-02-01 03:05:15 on dist-test-slave-gf59"
server_key: "03b056f9a2e4fd8cab7f85a2581082fb"
server_key_iv: "ecd8c52183a9b341e2230d6aa756fcc8"
server_key_version: "kuduclusterkey@0"
I20260201 03:05:15.094022 32470 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance:
uuid: "39fbe509bc164eb78223e3fbbc1f82b0"
format_stamp: "Formatted at 2026-02-01 03:05:15 on dist-test-slave-gf59"
server_key: "03b056f9a2e4fd8cab7f85a2581082fb"
server_key_iv: "ecd8c52183a9b341e2230d6aa756fcc8"
server_key_version: "kuduclusterkey@0"
I20260201 03:05:15.097822 32470 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260201 03:05:15.113790 32492 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260201 03:05:15.115029 32470 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260201 03:05:15.115168 32470 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/wal
uuid: "39fbe509bc164eb78223e3fbbc1f82b0"
format_stamp: "Formatted at 2026-02-01 03:05:15 on dist-test-slave-gf59"
server_key: "03b056f9a2e4fd8cab7f85a2581082fb"
server_key_iv: "ecd8c52183a9b341e2230d6aa756fcc8"
server_key_version: "kuduclusterkey@0"
I20260201 03:05:15.115316 32470 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260201 03:05:15.147521 32470 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260201 03:05:15.150449 32470 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260201 03:05:15.150650 32470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260201 03:05:15.151273 32470 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260201 03:05:15.152272 32470 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260201 03:05:15.152328 32470 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260201 03:05:15.152410 32470 ts_tablet_manager.cc:616] Registered 0 tablets
I20260201 03:05:15.152436 32470 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260201 03:05:15.164971 32470 rpc_server.cc:307] RPC server started. Bound to: 127.18.203.194:43519
I20260201 03:05:15.165388 32605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.203.194:43519 every 8 connection(s)
I20260201 03:05:15.166471 32470 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/data/info.pb
I20260201 03:05:15.170186 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:15.168757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.194:33641)
0201 03:05:15.169287 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:15.169411 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":126,"threads_started":1}
W20260201 03:05:15.170580 32606 heartbeater.cc:646] Failed to heartbeat to 127.18.203.254:33549 (0 consecutive failures): Network error: Failed to ping master at 127.18.203.254:33549: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
I20260201 03:05:15.171324 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:15.171081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.194:56147)
0201 03:05:15.171200 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:15.171253 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260201 03:05:15.171959 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:15.171742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.194:36807)
0201 03:05:15.171841 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:15.171888 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260201 03:05:15.172114 32606 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260201 03:05:15.176342 19247 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskyeWP68/build/debug/bin/kudu as pid 32470
I20260201 03:05:15.176446 19247 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskyeWP68/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769914928154323-19247-0/minicluster-data/ts-1/wal/instance
I20260201 03:05:15.176687 19247 external_mini_cluster.cc:1468] Setting key 299a7cd388ced7a68155af88723aa8d1
I20260201 03:05:15.895576 32457 heartbeater.cc:499] Master 127.18.203.254:33549 was elected leader, sending a full tablet report...
I20260201 03:05:15.896680 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:15.896189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.193:38107)
0201 03:05:15.896475 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:15.896558 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
W20260201 03:05:15.896981 32457 heartbeater.cc:646] Failed to heartbeat to 127.18.203.254:33549 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
I20260201 03:05:16.173250 32611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:16.172733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.194:36285)
0201 03:05:16.173040 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:16.173106 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260201 03:05:16.898272 32612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:16.897737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.193:34061)
0201 03:05:16.898067 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:16.898134 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":90,"threads_started":1}
I20260201 03:05:17.174533 32613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:17.174052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.194:53677)
0201 03:05:17.174355 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:17.174416 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":107,"threads_started":1}
I20260201 03:05:17.899717 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:17.899151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.193:59731)
0201 03:05:17.899481 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:17.899549 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":90,"threads_started":1}
I20260201 03:05:18.175715 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:18.175270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.194:45697)
0201 03:05:18.175538 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:18.175595 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260201 03:05:18.901021 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:18.900534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.193:45271)
0201 03:05:18.900836 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:18.900900 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
I20260201 03:05:19.176913 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:19.176460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.194:58369)
0201 03:05:19.176743 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:19.176804 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
I20260201 03:05:19.902336 32618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:19.901855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.193:55281)
0201 03:05:19.902166 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:19.902231 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":108,"threads_started":1}
I20260201 03:05:20.178344 32619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0201 03:05:20.177832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.203.254:33549 (local address 127.18.203.194:55905)
0201 03:05:20.178159 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0201 03:05:20.178231 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.203.254:33549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":116,"threads_started":1}
F20260201 03:05:20.214135 19247 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1769915120 (unix time) try "date -d @1769915120" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800004b2f) received by PID 19247 (TID 0x7f4319eafd80) from PID 19247; stack trace: ***
@ 0x7f4325b56980 (unknown) at ??:0
@ 0x7f432318cfb7 gsignal at ??:0
@ 0x7f432318e921 abort at ??:0
@ 0x7f43243fadcd google::LogMessage::Fail() at ??:0
@ 0x7f43243feb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f43243fa7cc google::LogMessage::Flush() at ??:0
@ 0x7f43243fbf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4327187cab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f43271849c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x555b6f5c492f kudu::SecurityITest::StartCluster() at ??:0
@ 0x555b6f5acc15 kudu::SecurityITest_TestEncryptionWithKMSIntegrationMultipleServers_Test::TestBody() at ??:0
@ 0x7f432540b0ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f43253ffbdb testing::Test::Run() at ??:0
@ 0x7f43253ffd9d testing::TestInfo::Run() at ??:0
@ 0x7f4325400377 testing::TestSuite::Run() at ??:0
@ 0x7f432540077c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f432540b60d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f43253ffe63 testing::UnitTest::Run() at ??:0
@ 0x555b6f5fcc7e RUN_ALL_TESTS() at ??:0
@ 0x555b6f5fbe6b main at ??:0
@ 0x7f432316fbf7 __libc_start_main at ??:0
@ 0x555b6f59406a _start at ??:0
2026-02-01 03:05:20.406 UTC [32188] FATAL: terminating connection due to unexpected postmaster exit
2026-02-01 03:05:20.406 UTC [32085] FATAL: terminating connection due to unexpected postmaster exit
2026-02-01 03:05:20.406 UTC [32080] FATAL: terminating connection due to unexpected postmaster exit
2026-02-01 03:05:20.407 UTC [32084] FATAL: terminating connection due to unexpected postmaster exit
2026-02-01 03:05:20.407 UTC [32081] FATAL: terminating connection due to unexpected postmaster exit
2026-02-01 03:05:20.408 UTC [32082] FATAL: terminating connection due to unexpected postmaster exit